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9TH  ANNUAL  SYSTEMS  ENGINEERING 


CONFERENCE 


Conference  Announcement 


October  23  -  26,  2006 


A  major  Conference  focusing  on 
improving  acquisition  and  performance 
of  Defense  programs  and  systems, 
including  net-centric  operations  and 
data/information  interoperability, 
system-of-systems  engineering,  and  all 
aspects  of  system  sustainment,  will  be 
convened  in  San  Diego,  CA,  October 
23-26,  2006.  This  Conference  is 
sponsored  by  the  National  Defense 
Industrial  Association,  Systems 
Engineering  Division,  with  technical 
co-sponsorship  by  IEEE  AES,  IEEE 
Systems  Council,  and  the  International 
Council  on  Systems  Engineering,  and 
is  supported  by  the  Office  of  Under 
Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics,  Director, 
Systems  and  Software  Engineering 
and  Office  of  the  DoD  Chief 
Information  Officer. 


Conference  Objectives 


This  conference  seeks  to  create  an  interactive  forum  for  Program  Managers,  Systems 
Engineers,  Chief  Scientists,  and  engineers  and  managers  from  the  government,  industry, 
and  academic  communities  whose  interests  converge  on  Defense  acquisition,  from 
capabilities  analysis  through  disposal.  This  Conference  will  provide  the  opportunity  to 
shape  policy  and  guidance  through  the  exchange  of  innovative  procedures  and  lessons 
learned  to  address  the  following  current  issues: 


Interoperability  &  Systems  Integration 
System-of-Systems  Engineering 
Life  Cycle  Systems  and  Program  Management 
Aging  Aircraft 

Life  Cycles  Systems  Management 
Sustainment  &  Upgrade  of  Legacy  Systems 
Application  of  Government  &  Industry  “Best 
Practices”  Yools,  Methodologies 
&  Technologies 

Capability  Maturity  Model  Integration  (CMMI) 
Integrated  Systems  Engineering,  Test, 

&  Supportability  Disciplines 
Network  Centric  Operations 
Application  of  DoD  Initiatives: 

•  Performance-Based  Business  Environment 

•  System  Safety 

•  Open  Systems 

•  Simulation-Based  Acquisition 

•  COTS  Integration 


Systems  Engineering  Effectiveness 
Modeling  &  Simulation 
Integrated  Risk  Management 
Performance  Based  Logistics 
Improved  Cycle  Times  for  Design, 
Manufacture  &  Repair  Process 
Improved  Mission  Readiness  & 
Systems  Availablility 
Systems  Engineering  Training  & 
Education 


Background 

The  Department  of  Defense  in  undertaking  a  major  transformation  of  our  military 
capability  in  response  to  the  new  world  environment  and  unforeseen  threats.  The 
ability  to  effect  this  transformation  can  only  be  realized  if  our  Defense  systems 
-  space,  air,  land,  sea,  and  under  sea  -  can  effectively  satisfy  mission  area  and 
capability  requirements  and  achieve  and  sustain  a  high  degree  of  interoperability, 
systems  integration,  systems  safety,  readiness,  and  availability.  We  believe  that 
the  greatest  opportunity  to  achieve  these  objectives  for  new  and  legacy  systems 
is  through  strong  technical  management  embodied  in  systems  engineering 
methodologies  and  processes,  on  the  part  of  both  industry  and  the  DoD. 

Strong  emphasis  on  systems  engineering  across  the  full  acquisition  life  cycle, 
from  concept  refinement  through  sustainment,  is  a  key  enabler  of  incremental 
development  and  evolutionary  acquisition.  The  Systems  Engineering  Conference  is 
an  annual  event  targeted  at  exploring  the  role  of  technical  planning  and  execution 
in  Defense  programs  and  systems  from  a  variety  of  perspectives,  academic  and 
pragmatic,  by  the  entire  Defense  systems  engineering  community. 
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contact :  Britt  Bommelje,  Associate  Director,  at 
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Hotel  Information 


Hyatt  Regency  Islandia 

1441  Quivira  Road 

San  Diego,  California  92109 

(619)  224-1234 


|  For  more  information  about  hotel  and  local  attractions,  please  visit  the  hotel  website  at:  httD://islandia.hvatt.com/hvatt/hotels/index.isD' 


Please  call  the  hotel  directly  in  order  to  make  your  reservation.  A  block  of  rooms  have  been  reserved  at  the 
Hyatt  Regency  Islandia.  In  order  to  ensure  the  discounted  NDIA  rate,  you  must  make  your  reservations  early, 
and  ask  for  the  NDIA  room  block.  Rooms  will  not  be  held  after  Friday,  September  29,  2006,  and  may  sell  out 
before  then.  Rates  are  also  subject  to  increase  after  this  date.  The  government  per  diem  rate  is  available 
only  to  active  duty  or  civilian  government  employees.  ID  will  be  required  upon  check-in.  Retired  military  ID’s  do 
not  qualify. 

Government  Rate:  $129.00*  Industry  Rate:  $169.00 

*(or  prevailing  government  per  diem  rate  at  the  time  of  the  Conference) 


Registration  Information 


•  EARLY* _ *  REGULAR  * _ *  LATE  » 


before  8/30/06 

from  8/30/06  to  10/6/06 

after  10/6/06 

Government/Allied/ 

Academia 

$660 

$725 

$800 

Industry  NDIA  Member 

$775 

$855 

$940 

Industry  Non-  NDIA 
Member 

$825 

$910 

$1,000 

Tutorial  Registration:  $200.00  -  This  registration  is  for  the  MONDAY  TUTORIAL  SESSION  ONLY,  and  it  is  IN  ADDITION 
TO  the  Conference  registration  fee 


To  register  online  for  this  Conference,  please  visit  the  following  link:  http://www.ndia.org/meetings/7870. 
Online  registration  will  close  on  October  6,  2006.  You  must  register  onsite  after  this  date.  You  can  also 
download  the  registration  form  from  the  NDIA  website  or  complete  the  form  contained  in  this  brochure.  Fax 
the  completed  form  to  703-522-1885  or  mail  to  Event  #7870,  National  Defense  Industrial  Association,  2111 
Wilson  Boulevard,  Suite  400,  Arlington,  VA  22201-3061.  Please  do  not  fax  or  mail  registration  froms  after 
October  6,  2006.  You  will  need  to  register  onsite  after  this  date.  Payment  MUST  be  made  at  the  time  of 
registration.  Registrations  will  not  be  taken  over  the  phone.  For  additional  questions,  contact 
Britt  Bommelje,  Associate  Director  at  703-247-2587,  or  via  email  at  bbommelie@ndia.org.  Please  see  page 
13  for  the  Conference  Registration  form. 

CANCELLATIONS  REMINDER:  Cancellations  received  before  August  30,  2006  will  receive  a  full  refund. 
Cancellations  received  from  August  30 , 2006  until  October  6,  2006  will  receive  a  refund  minus  a  cancellation 
fee  of  $75.00.  REFUNDS  WILL  NOT  BE  GIVEN  FOR  CANCELLATIONS  RECEIVED  AFTER  OCTOBER  6,  2006. 
Substitutions  are  welcome  in  lieu  of  cancellations. 
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Displayer  Information 


NDIA  invites  you  to  display  at  the  9th  Annual  Systems  Engineering  Conference.  An  area  will  be  available 
for  the  set-up  of  Corporate  or  Organization  displays  to  demonstrate  your  company/organization’s  capability 
in  Systems  Engineering  and  Design  expertise,  Logistics  Support,  Systems  Engineering  and  Design  Tools, 
Modeling  &  Simulation  Tools  &  Capability,  expertise  in  Test  &  Evaluation,  Reliability  Analysis  T  ools  &  Capability, 
Process  Improvement  Capability  &  Tools  including  CMMI  Appraisal  Services  &  Tools,  Engineering  Services  and 
related  items.  Displays  will  be  table-top  or  “pop-up”  style.  Allocated  display  space  will  be  10  ft  wide  by  6  ft 
deep.  Pipe  and  drape  is  not  provided.  Please  see  page  14  for  the  Display  Registration  Form. 

For  additional  information  on  displays,  please  contact  Britt  Bommelje,  at  703-247-2587  or  at 
bbommelie@ndia.org. 


General  Conference  Information 


Identification  Badges :  During  Conference  registration  and  check-in,  each  participant  will  be  issued  an 
identification  badge.  Please  be  prepared  to  present  a  picture  ID.  Badges  must  be  worn  at  all  conference 
functions. 

Conference  Proceedings:  Proceedings  will  be  available  on  the  web  through  the  Defense  Technical 
Information  Center  (DTIC),  and  will  be  available  two  to  three  weeks  after  the  Conference.  You  will  receive 
notification  via  e-mail  that  proceedings  are  posted  and  available  on  the  web. 

Promotional  Partnership  Opportunities:  Increase  your  company  or  organization’s  exposure  at  this  premier 
event  by  becoming  a  Promotional  Partner.  A  Promotional  Partnership  will  add  your  company  name  to  the 
back  cover  of  the  on-site  brochure  as  well  as  main  platform  recognition  throughout  the  conference,  signage  at 
all  events  including  the  opening  reception,  a  350-word  organization  description  in  the  on-site  brochure,  and  a 
hotlink  from  the  Conference  webpage  to  your  company  website.  For  more  information,  please  contact 
Sam  Campagna  at  703-247-2544  or  via  e-mail  at  scampagna@ndia.org. 

Attire:  Appropriate  dress  for  this  Conference  is  business  casual  for  civilians  and  class  B  uniform  for  military. 

ADA:  NDIA  supports  the  Americans  with  Disabilities  Act  of  1990.  Attendees  with  special  needs  should  call 
703-522-1820  and  refer  to  Event  #  7870  prior  to  October  6,  2006. 

National  Defense  Magazine:  Advertise  in  National  Defense  and  increase  your  company’s  exposure  at 
this  Conference!  National  Defense  will  be  distributed  to  the  attendees  of  this  Conference  and  all  other  NDIA 
Conferences.  For  more  information,  please  contact  Dino  Pignotti  at  703-247-2582  or  dpignotti@ndia.org. 

Inquiries:  For  questions  regarding  the  Conference,  please  contact  Britt  Bommelje,  Associate  Director  at 
703-247-2587  or  at  bbommelie@ndia.org. 

For  more  information  on  the  Conference,  or  to  register  online,  please  visit  the  Conference  website  at 
www.ndia.org/meetings/7870. 
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Conference  Agenda  --  At  a  Glance 

•Sun,  October  22,  2006  •  •  Sun,  October  22  •  •  Sun,  October  22  •  *Sun,  October  22  •  •  Sun,  October 

5:00  pm  -  7:00  pm  Registration  for  Tutorials  and  General  Conference 

(Tutorials  are  an  additional  $200.00  registration  fee) 


•Mon,  October  23,  2006  •  •  Mon,  October  23  •  •  Mon,  October  23  •  *Mon,  October  23  •  •  Mon, 
7:00  am  -  5:00  pm  Registration 

7:00  am  -  8:00  am  Continental  Breakfast  for  Tutorial  Attendees 

(Tutorials  are  an  additional  $200.00  registration  fee) 

8:00  am  - 12:00  pm  Tutorial  Tracks 

(Please  refer  to  page  7  for  Tutorial  Schedule) 

12:00  pm  - 1:00  pm  Buffet  Lunch  for  Tutorial  Attendees 

1:00  pm  -  5:00  pm  Tutorial  Tracks  Continued 

5:00  pm  -  6:00  pm  Reception  in  Display  Area  (Open  to  All  Participants) 

Regency  Annex 


•Tue,  October  24,  2006  •  •  Tue,  October  24  •  •  Tue,  October  24  •  *Tue,  October  24  •  •  Tue,  October 
7:30  am  -  5:00  pm  Registration 

7:30  am  -  8:30  am  Continental  Breakfast 

8:30  am  -  8:35  am  Introductions 

Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

8:35  am  -  8:45  am  Opening  Remarks 

Mr.  Bob  Rassa,  Director,  Systems  Supportability,  Raytheon;  Chair, 
Systems  Engineering  Division,  NDIA 

8:45  am  -  9:45  am  Keynote  Address 

HON  James  Finley,  Deputy  Under  Secretary  of  Defense,  Acquisiton  & 
Technology 

9:45  am  - 10:15  am  Break 

10:15  am  - 12:00  pm  Plenary  Session  :  Executive  Panel 

Moderator: 

Mr.  Mark  Schaeffer,  Director,  Office  of  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  Director,  Systems  and  Software 
Engineering 

Panelists: 

Mr.  Carl  Siel,  ASN/RDA  Chief  Engineer  (Acting) 

Mr.  Terry  Jaggers,  SAF/AQ,  Director,  SAF/AQR  (Science,  Technology 
&  Engineering)  (invited) 

Mr.  Doug  Wiltsie,  Assistant  Deputy  for  ASM,  HQDA,  OASA  (ALT)  (Invited) 
Mr.  Steven  Kapurch,  Chief  Systems  Engineer,  NASA  (Invited) 


Register  now!!  Visit  the  conference  website  at  http://www.ndia.org/meetings/7870  5 


Luncheon  Awards  Ceremony 


12:00  pm  - 1:30  pm 
Regency  Annex 

1:30  pm  -  5:00  pm 

5:00  pm  -  7:00  pm 
Wed,  October  25,  2006  •  • 
7:00  am  -  5:15  pm 
7:00  am  -  8:00  am 
8:15  am  - 12:00  pm 


12:00  pm  - 1:30  pm 
Regency  Annex 


Concurrent  Sessions  (Please  refer  to  pages  8-12  for  session 
schedule) 

Reception  in  Display  Area 

Wed,  October  25  •  •  Wed,  October  25  •  *Wed,  October  25  •  •  Wed, 
Registration 
Continental  Breakfast 

Concurrent  Sessions  (Please  refer  to  pages  8-12  for  session 
schedule) 

Luncheon  Speaker 

Mr.  Patrick  Michael  Kern,  Deputy  Assistnat  Secretary  of 
Defense,  NII/DoD  CIO  for  Enterprise  Wide  Systems  Engineering 


2:30  pm  -  5:30  pm 


Concurrent  Sessions  (Please  refer  to  pages  8-12  for  session 
schedule) 


•Thurs,  October  26,  2006  •  •  Thurs,  October  26  •  •  Thurs,  October  26*  •Thurs,  October  26  •  •  Thurs, 


7:00  am  -  3:00  pm 

Registration 

7:00  am  -  8:00  am 

Continental  Breakfast 

8:00  am  - 12:00  pm 

Concurrent  Sessions  (Please  refer  to  pages  8-12  for  session 
schedule) 

12:00  pm  - 1:00  pm 

Luncheon 

1:00  pm  -  3:00  pm 

Concurrent  Sessions  (Please  refer  to  pages  8-12  for  session 
schedule) 

3:00  pm 

Conference  Adjourns 

“The  Department  of  Defense  finds  this  event  meets  the  minimum  regulatory  standards  for  attendance  by  DoD 
employees.  This  finding  does  not  constitute  a  blanket  approval  or  endorsement  for  attendance.  Individual  DoD 
component  commands  or  organizations  are  responsible  for  approving  attendance  of  its  DoD  employees  based  on 
mission  requirements  and  DoD  regulations.” 


Register  now!!  Visit  the  conference  website  at  http://www.ndia.org/meetings/7870  6 
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12:00  pm  Lunch  in  the  Regency  Annex 


9th  Annual  Systems  Engineering  Conference 

Hyatt  Islandia,  San  Diego,  CA 
_ October  23-26,  2006  •  Event  #7870 _ 

3  Ways  to  sign  up:  1 .  Online  with  a  credit  card  at  www.ndia.org 

2.  By  fax  with  a  credit  card  —  Fax:  703-522-1885 
3.  By  mail  with  a  check  or  credit  card 


National  Defense  Industrial  Association 
2111  Wilson  Boulevard,  Suite  400 
Arlington,  VA 22201 -3061 
(703)  522-1820  •  (703)  522-1885  fax 

www.ndia.org 


o  Address  change  needed 


By  completing  the  following,  you  help 
us  understand  who  is  attending  our 
meetings. 


NDIA  Master  ID/Membership  # 

Social  Security  # 

(if  known — hint:  on  mailing  label  above  your  name) 

Prefix 

(last  4  digits  -  optional) 

(e.g.  RADM,  COL,  Mr.,  Ms.,  Dr.,  etc.) 

Name  First 

Ml  Last 

Military  Affiliation 

Nickname 

(e.g.  USMC,  USA  (Ret.)  etc.) 

(for  Meeting  Badges) 

Title 

Organization 

Street  Address 

Address  (Suite,  PO  Box,  Mail  Stop,  Building,  etc.) 

City  State 

Zip  Country 

Phone 

ext.  Fax 

E-Mail 

Signature* _ Date 

Preferred  way  to  receive  information 

Conference  information  o  address  above 

Subscriptions  o  address  above 

Alternate  Street  Address 
Alternate  Address  (Suite,  PO  Box,  Mail  Stop,  Building,  etc.) 

City _ State _  Zip _  Country _ 

*  By  your  signature  above  you  consent  to  receive  communications  sent  by  or  on  behalf  of  NDIA,  its  Chapters, 
Divisions  and  affiliates  (NTS A,  AFEI,  PSA,  NCWG,  WID)  via  regular  mail,  e-mail,  telephone,  or  fax.  NDIA,  its 
Chapters,  Divisions  and  affiliates  do  not  sell  data  to  vendors  or  other  companies. 


O  Alternate  (print  address  below)  O  E-mail 

O  Alternate  (print  address  below) 


Primary  Occupational 
Classification.  Check  ONE. 
oA.  Defense  Business/llndustry 
oB.  R&D/Laboratories 
o  C.  Army 
o  D.  Navy 
o  E.  Air  Force 
o  F.  Marine  Corps 
o  G.  Coast  Guard 
o  H.  DOD/MOD  Civilian 
o  I.  Gov’t  Civilian  (Non-DOD/ 
MOD) 

o  J.  Trade/Professional  Assn, 
o  K.  Educator/Academia 
o  L.  Professional  Services 
o  M.  Non-Defense  Business 
oN.  Other _ 

Current  Job/Title/Position. 

Check  ONE. 
oA.  Senior  Executive 
o  B.  Executive 
o  C.  Manager 
o  D.  Engineer/Scientist 
o  E.  Professor/Instructor/Librarian 
o  F.  Ambassador/Attache 
o  G.  Legislator/Legislative  Aide 
o  H.  General/Admiral 
o  I.  Colonel/Navy  Captain 
oJ.  Lieutenant  Colonel/ 
Commander/Major/ 
Lieutenant  Commander 
o  K.  Captain/Lieutenant/Ensign 
o  L.  Enlisted  Military 
o  O.  Other _ 

Year  of  birth _ 

(Optional) 


Registration  Fees 


Government/ Academia 
Industry/NDIA  Member 
Industry/Non-Member 


Early 

before 

08/30/06 

Regular 

8/30/06- 

10/06/06 

Late 

after 

10/06/06 

o  $660 

0  $725 

o  $800 

o  $775 

o  $855 

o  $940 

o  $825 

o  $910 

o  $1000 

Tutorial  Registration  -  MONDAY  ONLY  □  $200 
Tutorial  registration  is  for  the  Monday  Tutorial  Session 
ONLY,  and  it  is  in  addition  to  the  Conference  Registration 
Fee 


Payment  Options 

o  Check  (payable  to  NDIA) 
o  Cash 

o  Government  PO/Training  Form  # 
oVISA 
o  MasterCard 
o  American  Express 
o  Diners  Club 

If  paying  by  credit  card,  you  may  return  by  fax  to  (703)  522-1885. 
Credit  Card  Number 


□□□□□□□□□□□□□□□□ 

Exp.  date 


Cancellations  received  before  August  12,  2006  will  receive 
a  full  refund.  Cancellations  received  between  August  12, 
2006  and  October  6,  2006  will  receive  a  refund  minus  a 
cancellation  fee  of  $75.  No  refunds  for  cancellations 
received  on  or  after  October  6,  2006. 

Substitutions  welcome  in  lieu  of  cancellation. 


Signature  Date 

Questions?  Contact  Britt  Bommelje,  Associate  Director 

(703)  247-2587  email:  bbommelje@ndia.org 
Mail  to:  NDIA,  Event  #7870 

2111  Wilson  Boulevard,  Suite  400 
Arlington,  VA  22201 
Fax  to:  (703)522-1885 
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Registration  for  Displays,  Event  #  7870 
9th  Annual  Systems  Engineering  Conference 
October  23-26,  2006 
Hyatt  Islandia 
San  Diego,  CA 


Name _ 

Title _ 

Company  Name - 

Division/Dept. - 

Address - 

City/State/Zip _ 

Phone _ F  ax _ 

E-mail _ 

Display  Requirements: 

All  displays  must  be  of  the  simple  table-top/pop-up  style  standards.  Space  per  display  shall  not  exceed  10  ft.  wide  by  6  ft.  deep. 
Minimal  hardware  to  be  utilized  (computer  systems  for  demonstrations  are  OK).  No  formal  decorating  company  is  involved. 
Companies  must  bring  their  own  displays  and  plan  to  do  their  own  set-up.  Standard  2.5  x  6  ft.  draped  folding  tables  and  chair 
will  be  provided  for  each  display  space.  No  other  props  or  setups  (pipe  &  drape,  plants,  etc.)  will  be  utilized. 

Display  Hours: 

Displays  are  to  be  set  up  by  4:00  PM  October  23,  2006  and  should  remain  in  place  until  after  the  mid-morning  break  on  October 
26,  2006.  Displays  must  be  removed  by  4:00  PM  October  26,  2006. 

Cost:  Displays  (includes  one  free  registration  and  electrical  hook-up  for  display):  $  1,000 

Display  Rules  &  Regulations: 

1.  If  NDIA  should  be  prevented  from  holding  the  conference  for  any  reason  beyond  NDIA’s  control  (such  as,  but  not  limited  to, 
damage  to  the  building,  riots,  strikes,  acts  of  government,  or  acts  of  God)  or  if  a  displayer  cannot  occupy  the  assigned  display 
space  due  to  reasons  beyond  NDIA’s  control,  then  NDIA  has  the  right  to  cancel  the  conference  or  any  part  thereof,  with  no  fur¬ 
ther  liability  to  the  displayer  other  than  a  refund  of  display  space  fee,  less  a  proportionate  share  of  the  conference  cost  incurred. 

2.  Neither  the  management  of  the  host  facility  nor  NDIA  shall  be  liable  for  the  damages,  loss  or  destruction  to  the  displays  by 
reason  of  fire,  theft,  accident  or  other  destructive  causes.  Displayer  shall  lease  space  at  his  sole  risk.  Neither  the  management 
of  the  host  facility,  NDIA  nor  any  of  their  agents,  servants  or  employees  will  be  accountable  or  liable  for  accidents  to  displayers, 
their  agents  or  employees. 

3.  The  displayer  shall  be  liable  to  the  host  facility  and/or  NDIA  for  any  damage  to  the  building  and/or  the  furniture  and  fixtures 
contained  therein  which  shall  occur  through  acts  or  omissions  of  the  displayer. 

4.  Displayer  assumes  the  entire  responsibility  and  hereby  agrees  to  protect,  indemnify,  defend  and  hold  harmless  NDIA,  the 
host  facility,  their  officers,  employees,  and  agents  against  all  claims,  losses  and  damages  to  persons  and  property,  governmental 
charges  or  fines,  and  attorney’s  fees  arising  out  of  or  caused  by  displayers  installation,  removal,  maintenance,  occupancy  or  use 
of  the  display  premises  or  any  part  thereof,  including  any  outside  display  areas. 

5.  Displayer  acknowledges  that  NDIA  does  not  maintain  and  is  not  responsible  for  obtaining  insurance  covering  displayer ’s 
property.  Displayers  are  advised  to  obtain  business  interruption  and  property  damage  and  loss  insurance  to  cover  such  occur¬ 
rences. 

Send  this  form  with  payment  for  display  to: 

Britt  Bommelje,  Associate  Director,  National  Defense  Industrial  Association,  2111  Wilson  Boulevard,  Suite  400,  Arlington,  VA 
22201-3061,  Phone:  (703)247-2587,  Fax:  (703)522-1885,  E-mail:  bbommelje@ndia.org. 

Deadline  for  sign-up  is  Friday,  October  13,  2006  .  Make  checks  payable  to  NDIA  -  Event  #  7870. 


O  Check  (payable  to  NDIA  -  Event  #7870) 

^Jvisa  |  |  Diner’s  Club  |  |  Mastercard  |  |  Amex 

Credit  Card  # _ Exp.  Date _ 

Authorized  Signature _ 
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Ride  the  Giant  Dipper  Rollar  Coaster  at 
Mission  Beach 


Experience  a  San  Diego  Charger 
Football  game  -  Sunday, 

October  29  1:05  pm 

vs.  St.  Louis  Rams  Tour  USS  Midway  in  San  Diego’s 

Port 


For  more 

information  on  what  to  do  in  San  Diego  please  visit 

www.sandiego.org 


2111  Wilson  Blvd. 

Suite  400 

Arlington,  VA  22201-3061 
www.ndia.org 


9th  Annual  Systems  Engineering  Conference 
“Focusing  on  Improving  Performance  of  Defense  Systems  Programs” 

October  23-26,  2006 
San  Diego,  CA 


Defense  View 
on 

Considerations 
for  System  of  Systems  SE 


25  October  2006 


Kristen  Baldwin 
Systems  and 
Software  Engineering 
Office  of  the  Under 
Secretary  of  Defense 
(AT&L) 


Dr.  J  udith  Dahmann 
The  MITRE  Corporation 


Robin  Gu I  lifer 
Systems  and 
Software  Engineering 
Office  of  the  Under 
Secretary  of  Defense 
(AT&L) 


Purpose  and  Overview 


•  Purpose  of  Presentation 

-  Provide  an  update  on  AT&L  SE  activities  to  support  capabilities  and 
systems  of  systems 

•  Overview  of  Topics 

-  Background 

-  Recent  events 

•  I NCOSE,  QDR  Portfolio  Test  Cases,  DoD  SW  Strategy  Summit 

-  New  initiative 

•  DUSD  (A&T)  directed  OSD- led  effort  to  develop  and  publish  System-of- 
Systems  (SoS)  Systems  Engineering  Guide 

-  Way  Ahead 
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DAB  Context  Slides 


MORS  Capabilities- Based  Planning  (CBP) 
Conference  Oct  2004 


Missile 

Defense 

Agency 


Strategic 
Planning 
Guidance 
CBP  Study 
J  uly  2004 


o 


I  ntegrated 
Air  &  Missile 
Defense 
Capability 
Area 

Review  DAB 
March  2005 


System 
&  SW 
Tech 
Conf 
April 
2005 


1st  Annual 
SoS  SE 
Conference 
J  une  2005 

yy  Stevens  I  nst 
|  }  SoS  SE 

Workshop  I 
Oct  2005 


NDI A 
SE  Conf 
Oct  2005 

o 


Stevens  I  nst 
SoS  SE 
Workshop  1 1 
Jan  2006 


QDR 

2006 


2004 


t 


DOD 

5000 


JCI DS 
3170 


2005  t 


NDI  A 
JCI  DS/ 
M&S 

Conference 


2006 


PA&E 

Costing  SoS 
Study 


AF  Science 
Board 
SoS  SE 
Study 


Naval 

Capabilities 

Evolution 

Process 

(NCEP) 


Joint 

I  ntegrated 
Air  and 
Missile 
Defense 
(J  I  AMD) 
Summit 


Decision 
Support 
Center  J  oint 
Distributed 
SE  and  Test 
Study 


Roadmaps  &  Capability  Area  Reviews 


->  Tri-Chair  Concept  Decisicgis 


I NCOSE  Panel  on  System  of  Systems 

J  uly  2006 


•  One  of  a  number  of  events  addressing  issues  of  SOS 

•  Series  of  presentations  from  academia  and  industry 

•  Quotable  quotes 

-  ‘There  is  no  nice  line  between  Systems  and  SoS” 

-  “There  is  no  difference  between  SE  for  systems  and  SOS...” 

-  “There  is  a  simply  a  need  for  better  requirements  management  for 
SoS...” 

-  “Thinking  that  traditional  SE  methods/techniques  are  sufficient  for 
SOS  is  dangerous..” 

-  “Standard  SE  applies  but  requires  extensions” 

-  “Only  difference  is  no  one  in  control  in  a  SOS....” 

-  “Nothing  is  new.  Any  system  that  has  sub-systems  is  a  SoS.  We 
have  been  doing  this  forever.” 


Wide  range  of  perspectives  on  SOS  and  SE  today 
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Defense  Considerations  for  SoS 


•  Scale 

-  Size  of  defense  enterprise  makes  a  single  integrated  architecture  infeasible 

•  Ownership/  Management 

-  I  ndividual  systems  are  owned  by  the  military  component  or  agencies,  introducing 
constraints  on  management  and  SE 

•  Legacy 

-  Given  defense  budget  projections,  current  systems  will  be  part  of  the  defense  inventory 
for  the  long-term  and  need  to  be  factored  into  any  approach  to  SOS 

•  Changing  operations 

-  Changing  threats  and  concepts  mean  that  new  (ad  hoc)  SoS  configurations  will  be  needed 
to  address  changing,  unpredictable  operational  demands 

•  Criticality  of  SW 

-  SOS  typically  focus  on  integration  across  systems  through  cooperative  or  distributed 
software 

•  Role  of  network 

-  Conceptually  DOD  SoS  will  be  network  based;  budget  and  legacy  challenges  of  budgets 
and  legacy  mean  uneven  implementation 
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QDR  I  mplementation 
SE  for  Capability  Portfolio  Test  Cases 


...  there  is  a  need  for  systems  engineering  support  to 
ensure  that  the  set  of  capability  solutions  -  including 
legacy,  planned,  and  programmed  efforts  -  is  coordinated 
so  as  to  maximize  the  solutions'  effectiveness  and  ensure 
their  timely  delivery  to  the  warfighter... 

Systems  engineering  will  provide  the  technical  base  for 

selecting  components  of  the  systems  needed  to  support 
portfolio  objectives,  for  identifying  the  technical  aspects  of 
the  of  those  systems  critical  to  meeting  the  larger  portfolio 
capability  goals,  and  for  defining  and  assessing  the  end- 
to-end  performance  of  the  system  of  systems... 


•  Quadrennial 
Defense  Review 
directed  DOD 
move  towards  a 
portfolio  approach 
to  force 
development 
based  on  J  oint 
capabilities 


. . .  engineering  of  the  systems  will  remain  the 
responsibility  of  the  program  managers  or  components. . . 

system  of  systems  engineering  function  will  address 
technical  aspects  of  design,  configuration,  and  system 
integration  that  are  critical  to  meeting  joint  capability 
objectives... 

Deputy  Secretary  of  Defense  Capability  Portfolio 
Management  Test  Case  Guidance,  14  Sept  2006 


•  Four  test  cases 
have  been 
initiated 

•  I  n  each  case  SE  is 
seen  as  a  portfolio 
level  function 


Defense  SW  Strategy  Summit 

October  2006 


Top  Software  Issues  MPlTl 

Opportunities  for  Improvement 

1.  The  impact  of  requirements  upon  software  is  not  consistently 
quantified  and  managed  in  development  or  sustainment. 

2.  F  undamental  system  engi  neering  decisio  ns  are  made  without  ful  1 
participation  of  software  engineering. 

3.  Software  life-cycle  planning  and  management  by  acquirers  and 
suppliers  is  ineffective. 

4.  The  quantity  and  quality  of  software  engineering  expertise  is 
insufficient  to  meet  the  demands  of  government  and  the  defense 
industry. 

•  Development  environments  for  net-reliant 
embedded  systems  must: 

-  Readily  embrace  emerging  data  and  knowledge 
management  strategies 

-  Automatically  facilitate  and  assess  interoperability  and 

nrntnrnl  imnfprnpntfltinn 

-  Address  system- of- systems  design 

■  Prop erties-i n-th e-la rg e ,  composeability,  security 

5.  Traditional  software  verification  techniques  are  costly  and  j 

ineffective  for  dealing  with  the  scale  and  complexity  of  modern 
systems. 

-  AooommooaLe  uaLa  ariu  luriaioriai  u r ice r Lai ri lies 
associated  with  ad-hoc  networks  and  transient 
application  relationships 

*  System-of-Systems  Verification 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure 
execution  of  complex  software  in  distributed  environments. 

7.  Inadequate  attention  is  given  to  total  lifecycle  issues  for 

COTSAIDI  impacts  on  lifecycle  cost  and  risk. 

hDkf-  rrtdu-p-u  E bhbk:*  -Tif-  i¥f  kihM  i 

Viracn  Pit 

QDR  highlights  need  for  Systems  of  Systems 

Strategic  Initiatives 


Focus  Area 

|  Initiative 

Process 
and  People 
Focus 

1.  Acquisition  Process  Improvement 

Foster  migration  to  a  modei-b  ased  approach  for  continuously  impro  ving  the 
software  and  software  iriensive  systems  across  the  Amy 

2.  Measurement 

increase  the  visibility  of  software  metrics  in  system  management/deveiopment 

3.  Training  &  Education 

Prepare  Army  leadership,  Program  Managers,  and  Staffs  to  meet  the  technology 
and  management  challenges  inherent  to  the  acguisition  of  software  intensive 
systems 

Technology 

Focus 

4.  Architecture 

Inst&iiionalize  knowledge  and  practice  of  software  architecture  to  improve  guaiity, 
lower  costs  and  reduce  c.vc.le  times  for  software  and  software  iriensive  sv.tems 

5.  System  of  Systems  Integration 

Develop  strategies  and  technigues  to  identify  and  resolve  issues  in  meeting 
system  of  systems,  net-centric,  capability-based  reg  uire  ments  for  interoperability 
and  point  war  fightin^^^ 

Future 

Focus 

6.  Analysis  &  Strategic  Planning 

In'restigate  newcha llenges  and  opportunities 

•  Purpose 

-  Focus  efforts  on 
top  DOD  SW 
issues 

•  Panels  & 
workgroups 

-  PEOs,  SEs,  sw 
experts 

•  Recurring  topic 
of  SOS 

-  SW  Challenges 

-  PEO  and  Service 
initiatives 

-  Research  efforts 


SW  is  a  key  element  of  SOS 
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Profiling  the  Context  for  SE:  One  Model 


System 
Context 


Strategic 

Context 


Stakeholder 
Context 


Implements  tion 


Context 


Typical  program  domain 

Traditional  systems  engineering 

Chief  Engineer  inside  the  program; 
reports  to  program  manager 

Transitional  domain 

Systems  engineering  across 
boundaries 

Work  across  system/program 
boundaries 

Influence  vs  authority 

Messy  frontier 

Political  engineering  (power, 
control...) 

High  risk,  potentially  high  reward 
Foster  cooperative  beha  vior 


Source:  Kenee  Stevens 

MITRE 


©  2006  Hie  M  ITRE  Corp  oratio  n.  All  rights  rese  rved 


SoS  SE  Guide 


•  DUSD  (A&T)  directed  OSD- led  effort  to  develop  and  publish 
System- of- Systems  (SoS)  Systems  Engineering  Guide 

•  Purpose 

-  Leverage  current  experience  to  support  ongoing  efforts  to  develop, 
field  and  sustain  SoS 

-  Focus  on  technical  aspects  of  SE  applicable  across  SOS  management 
constructs 

•  Version  1 

-  6  month  effort  addressing  areas  of  agreement  across  the  community 

•  Audience  -  Program  Managers  and  Lead/Chief  Engineers 

•  Development  Participants 

-  Lead  by  AT&L  Systems  and  Software  Engineering 

-  I PT  with  representatives  from  Services  and  Agencies 

-  Stevens  I  nstitute  is  the  integrating  author 


Draft  (Version  .1)  of  Guide  is  now  out  for  review 
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The  Guide  Addresses  . . .. 


•  Definitions 

•  Scope 

•  Characteristics  of  the  SOS  environment 

•  I  llustrative  DOD  use  cases 

•  Challenges  and  approaches  to  SE  processes 
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Definitions 
Draft  SOS  SE  Guide 


System 

An  integrated  composite  of  people,  products,  and  processes  that  provide 
a  capability  to  satisfy  a  stated  need  or  objective 

Mil-Std  499B 


System  of  Systems 

A  set  or  arrangement  of  systems  that  results  when  independent  and 
useful  systems  are  integrated  into  a  larger  system  that  delivers  unique 
capabilities 

DoD  Defense  Acquisition  Guide,  System  of  Systems  Engineering 

System  of  Systems  Engineering 

Planning,  analyzing,  organizing,  and  integrating  the  capabilities  of  a  mix 
of  existing  and  new  systems  into  a  SoS  capability  greater  than  the  sum  of 
the  capabilities  of  the  constituent  parts 

DoD  Defense  Acquisition  Guide,  Chapter  4 
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Scope 

Spectrum  Of  I  ssues 


Current  Systems 
Engineering 
Practices 


System 

Context 


Strategic 
£/)U.  Context 


Stakeholder 

Context 


Implementation 
Context 


Focus  of 
I  nitial  Version  of 
SOS  SE  Guide 


Challenges 


SoS  cases  under  the  well-defined  circumstances  where  there  is 

(1)  defined  user  need 

(2)  resources  designated  to  address  the  need  and 

(3)  someone  has  the  responsibility  to  address  the  need 


12 


Draft  SoS 
Characterizing  the 


Guide 

Environment 


Community  I  nvolvement: 
Stakeholders,  Governance 

System:  stakeholders  generally 
committed  only  to  the  one  system 

SoS:  stakeholders  more  diverse; 
stakeholders  from  each  system  will  have 
some  interest  in  the  other  systems 
comprising  the  SoS 


k  d 


Employment  Environment:  Mission 
Environment,  Operational  Focus 

System:  mission  environment  is  relatively 
stable,  pre-defined,  and  generally  well-known; 
operational  focus  is  clear 

SoS:  emphasis  on  multiple  missions, 
integration  across  missions,  need  for  ad  hoc 
operational  capabilities  to  support  rapidly 
evolving  mission  objectives 


Implementation:  Acquisition/ Test 
Ana  Validation,  Engineering 

System:  aligned  to  ACAT  Milestones,  specified 
requirements,  a  single  DoD  PM,  SE  with  a 
Systems  Engineering  Plan  (SEP),  test  and 
validating  the  system  is  possible 

SoS:  multiple  system  lifecycles  across 
acquisition  programs,  involving  legacy  systems, 
developmental  systems,  and  technology  insertion 
with  multiple  DoD  PEOs,  PMs  and  operational 
and  support  communities;  testing  is  more 
difficult,  and  test  and  validation  can  be 
distributed  and  federated 


System 
Context 


Strategic 

Context 


d,: 

Stakeholder 
Context 


Ac&&***  Implementation 


Context 
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Defense  Experience  With  SoS  SE 
Several  Examples 


New  Development 

-  Future  Combat  System  (FCS)  and 

I  nteg rated  Deepwater  System  (I  DS) 

Development  of  New  Capability  by 
I  nteg  rating  Current  System 

-  Army  Battlefield  Command  System 
(ABCS) 

Mixed  System  Maturity  Levels 

-  Naval  I  ntegrated  Fire  Control-Counter 
Air  (NIFC-CA)  and  Single  Integrated 
Air  Picture  (SIAP) 

Sustaining  Capabilities 

-  Stryker  Brigade  Combat  Team  (BCT) 

IT/ Business 

-  Commissary  Advance  Resale 
Transaction  System  (CARTS) 


Draft  SOS  SE  Guide 

Challenges  of  SoS  for  SE  Processes 


Interfax 


CcmiSflu^iwn. 

Mnnjc.cmcnr 


Technical 

Aivtsywni 


TECHNICAL  PROCESSES 


Transition 


ftequirtowrls 

Owtgpmcnl; 


Validaiie-i 


Logk-il 

Jknabrift 


VcnHcaLan 


Dtdgn 

SoMipn 


Int&gnlion 


From  Chapter  4  Defense  Acquisition  Guide 


Technical  and  Technical  Management 
Processes  for  SE 

•  I  dentify  implications  of  SOS  for  each 
process 

•  Challenges  these  pose  for  the  SE 

•  Approaches  to  address  the  challenges 

Processes  apply,  but  the  SOS 
environment  affects  approaches, 
methods  and  tools  needed  by  SE 

•  More  collaboration,  less  top  down 

•  More  complexity  to  accommodate 
requirements,  approaches  and  tools 
used  by  constituent  systems 

•  Balance  between  roles  of  SOS  SE  and 
the  system  SE 

•  More  need  for  experimentation  to 
determine  ways  to  employ  existing 
systems  and  to  discover  effects  of 
combined  systems 
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Next  Steps 


Challenges 
Version  2  and  Beyond 
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NDIA 

9th  Annual  Systems  Engineering  Conference 

San  Diego ,  California 
October  23-26,  2006 


Implementing  an  Effective 
Peer  Review  Process 


Jeanne  Balsam 
Jean  Swank 
Lee  Sheiner 

Electronic  Systems  Laboratory 
Georgia  Tech  Research  Institute 
Georgia  Institute  of  Technology 


Georgia  [^©©©sHrdhi 
Tech  M  Oirassuto'S® 
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Georgia  Tech  Research  Institute  (GTRI)  Overview 


•  Unit  of  the  Georgia  Institute  of  Technology 

•  1200+  employees 

•  70%  of  research  employees  hold  advanced  degrees 

•  Wide  variety  of  products 

•  Customers  include  federal  and  state  government; 
and  industry 

•  Competitively  bid  projects  range  greatly  in  size  and 
duration 

•  More  Info:  http://www.qtri.qatech.edu/ 


■ 


■aV 


Outline 


•  Purpose  of  peer  review 

•  Formalize  the  peer  review  process 

•  Plan  peer  reviews 

•  Apply  the  process  of  a  peer  review 

•  Secondary  benefits  of  peer  reviews 


Georgia  [^©©©sHrdhi 
Tech  M  Oirassuto'S® 
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What  is  a  Peer  Review? 


“The  review  of  work  products  performed 
by  peers  during  development  of  the 
work  products  to  identify  defects  for 
removal.” 


-  CMMI  Guidelines  for  Process  Integration  and  Product  Improvement 

(Addison  Wesley,  2003,  page  622) 


Georgia  [^©©©sHrdhi 
Tech  M  Oirasrffc'O® 
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Why  Do  Peer  Reviews? 


•  Verify  the  work  product  meets  requirements 

•  Identify  defects  or  problems  early  in  the  life-cycle 


•  Gain  confidence  in  work  products 

*  Reduce  risk 


Georgia 

Tech 


[^©©©SHrdhi 

Oirasrffc'a® 


An  Informal  Peer  Review 


“Does  this  seem  right  to  you?” 


Georgia 

Tech  U  Q[ra®SB,a(ui'S® 
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An  Inappropriate  Peer  Reviewer 


“Farmer  Bob,  does  this  seem  right  to  you?” 


Georgia 

Tech 


[^©©©SHrdhi 

Oirasrffc'a® 
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Why  Do  We  Need  Formalized  Peer 

Review  Processes? 


A  formalized  process  helps  ensure: 

•  Peer  reviews  are  taking  place 

•  The  right  products  are  being  peer  reviewed  at  appropriate  times 

•  Adequate  resources  are  planned  and  allocated  for  peer  reviews 

•  The  right  reviewers  are  being  selected 

•  The  reviewers  are  prepared  adequately 

•  Defects  are  being  recorded 

•  Defects  are  tracked  to  closure 

II I  I  I 


Georgia  [^©©©sHrdhi 
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Document  the  Peer  Review  Process 


•  Types  of  reviews 

•  What  to  review  in  each  phase 

•  Planning 

•  Conducting 

•  Closing 


Georgia  [^©©©sHrdhi 
Tech  M  Oirassuto'S® 
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Peer  Review  Types 


Desk  Check 

•  Single  producer  and  single  reviewer 

•  Cheapest,  least  effective  review 

Round  Robin 

•  Single  producer  and  at  least  two 
reviewers 

•  Reviewers  examine  work  product 
sequentially 

•  A  single  defect  log  is  used 

•  Moderator  verifies  defects  are 
corrected 


Structured  Walkthrough 

•  One  or  more  producers,  at  least  two 
reviewers,  a  moderator,  and  a 
recorder 

•  All  participants  meet  after  reviewers 
have  prepared 

•  More  expensive  and  effective  than  a 
Round-Robin 

Formal  Inspection 

•  Roles  and  format  similar  to 
Structured  Walkthrough 

•  Outside  experts  participate 

•  Advanced  preparation  is  extensive 
and  required 

•  Most  expensive  and  effective 
review  type 


Georgia  [^©©©sHrdhi 
Tech  M  IhinttlHtaiito 


ImplementingAnEffectivePeerReviewProcess  - 10 


What  to  Review 


Requirements 

Design 

Implementation 

*  Critical  components 

*  Complex  components 

*  New  employee’s  work 

<  New  technology  or  platform 
Test  Plans 


Georgia  [^©©©sHrdhi 
Tech  M  Oirasrffc'O® 
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Plan  Peer  Reviews 


•  Determine  what  will  be  peer  reviewed 

•  Determine  when  it  will  be  peer  reviewed 

•  Provide  adequate  budget  for  peer  reviews 

•  Plan  for  critical  reviewers 

•  Plan  for  appropriate  facilities 

Georgia  [^©©©sHrdhi 
Tech  M  Oirassuto'S® 
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Applying  the  Process 


Georgia 

Tech 


[^©©©SHrdhi 

Oirasrffc'a® 


Georgia 

I  ©ffTechtraoDogj^ 


Prepare  for  Peer  Reviews 


*  Choose  reviewers 

*  Schedule  meeting 

*  Prepare  review  and  reference 
materials 


Georgia  Jl  I 

Tech  |  Q[ra®SD'S(La'S® 
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Choosing  Reviewers 


•  Knowledgeable  and 
trained 

•  Some  project-independent 
reviewers  are  desirable 

•  Non-management,  unless  special  circumstances 
require  a  manager’s  participation 

•  Committed  to  adequately  prepare 


Georgia  [^©©©sHrdhi 
Tech  M  Oirassuto'S® 
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Scheduling  the  Meeting 


•  Allow  the  reviewers  adequate  time  to  prepare 
defect  logs 


Define  clear  objectives  regarding  the  amount 
of  time  (min/max)  for  the  review  preparation 

Limit  meeting  time  to  two  hours 

Ideally  choose  a  location  with  a  networked 
computer,  overhead  projector,  and  access  to 
configuration  management  system 


Georgia 

Tech  U  □iraattB'atyi'S® 


ImplementingAnEffectivePeerReviewProcess  - 16 

Review  and  Reference  Materials 


Review  materials  must  be  under 
version  control 

Provide  controlled  defect  logs  to 
reviewers 

Identify  location  and  version  of  all 
review  materials 

Provide  reference  materials 


Georgia  [^©©©ardh) 
Tech  M  OiraattB'atm'S® 


I  implementing  An  Effe£ 


Georgia  OirD®,aD'iMto 
®ffTech[ra®D®gj^ 


Preceding  the  Peer  Review 


•  Verify  producer  has  distributed 
product  for  review 

•  Verify  that  reviewers  are 
prepared 

•  Tabulate  all  the  defects  into  a 
summary  log 


Georgia 

Tech 


[R3®®@srdli] 

Oirasrffc'a® 
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Conducting  the  Meeting 


*  Walk  through  the  work  product  in  its  entirety;  don’t  just 
look  at  the  tabulated  defects 

*  Ideally  -  use  a  projector  so  that  everyone  can  see  how 
defects  are  recorded 

*  Gain  consensus  during  the  review  of  the  type,  severity 
and  disposition  of  each  defect 

*  Identify,  but  don’t  try  to  fix  the  defects 

*  Determine  if  re-review  is  necessary 


Georgia  [^©©©sHrdhi 
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Closing  the  Peer  Review 


•  Put  peer  reviews  on  the  list  of  project  deliverables 
so  that  closing  them  won’t  fall  through  the  cracks 

•  Close  out  defects  or  write  a  change  request 

•  Re-review  if  necessary 

•  Require  project  director  and 
quality  engineer  signature  to 
close  the  review 


Georgia  [^©©©ardh) 
Tech  M  OiraattB'atm'S® 


II I  I 


ImplementingAnEffectivePeerReviewProcess  -  20 


Secondary  Benefits 


•  Create  mini-milestones  for  work  products 

•  Jump-start  team  communication 

*  Product  quality  increases  when  the  author  knows  it 
will  be  reviewed 

*  An  opportunity  for  team  building  within  the  project  - 
everyone  has  to  be  reviewed  and  act  as  a  reviewer 


Georgia  [^©©©sHrdhi 
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More  Secondary  Benefits 


•  Leverage  team  member  skills 

•  Teach  junior  engineers  “It’s  OK  to  criticize  senior 
people’s  work” 

•  Exposes  junior  engineers  to  direct  tutelage  from 
experts 

•  Expose  reviewers  from  outside  the  project  team  to 
new  ideas,  and  vice-versa 


Georgia  [^©©©sHrdhi 
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Contact  Information 


•  Jeanne  Balsam 

•  ieanne.balsam@qtri.qatech.edu 

•  Jean  Swank 

•  iean.swank@qtri.qatech.edu 

•  Lee  Sheiner 

•  lee.sheiner@qtri.qatech.edu 

•  More  Info  about  GTRI: 

http://www.qtri.qatech.edu/ 
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NDIA  Systems  Engineering  Conference,  Oct  24  -  26  2006 

San  Diego,  CA 

Raymond  F.  Beach 

NAVAIR  ASSOCIATE  FELLOW 
EO/RADAR  BRANCH  HEAD 


AGENDA 


BACKGROUND 

APPROACH 

SPECIFICATION  vs.  REALITY 
FACTORS  AFFECTING  T&E 
DEVELOPMENT  TEST  AND  EVALUATION 
FUTURE  TRENDS 


NAVAL  AVIATION  T E £ H M UL UU'LU 


QUESTIONS 


BACKGROUND 


WHAT  IS  SYSTEMS  ENGINEERING? 


•  ...  Focuses  on  methods  to  solve  problems,  not  the  solution  of  the 
problem.... 

#...  Specifications  and  performance  metrics . 

#....  Optimization  methods  in  presence  of  constraints...  . 

•  Modeling  and  Simulation 
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APPROACH 


WHAT  THE 

WARFIGHTER 

WANTS/NEEDS 


^\r 


SYSTEMS 

ENGINEERING 


> 


WHAT  THE 

WARFIGHTER 

GETS 
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PERCEIVED  ACQUISITION  PROCESS 


How  the 

Fleet  described  it 


How  the  requirement 
was  understood 


How  the  contractor 
designed  it 


How  the  programmer 
wrote  it 


How  OPNAV  describee 
it 


How  the  project  was 
documented 


What  SYSCOM 
installed 


How  the  Government 


htlloH 


How  the  helpdesk 

ci  innnrtorl  it 


What  the  Sailor 
really  needed 
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T&E  PROCESS 
(CONCEPTUAL) 


Concept/Design  to  meet  Need 


Build  System 


Actual  testing 


Vision  for  Need  (Warfighter) 
Articulate  need  (Service  to  Congress) 


SPECS  to  Testers  (PM A  to  us) 


ETHER 


Pound  Pavernent  for  Support 
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SPECIFICATION  vs.  REALITY 


•  YARDSTICK  OF  PERFORMANCE  DURING  DT 

•  WHAT  IS  NEEDED  vs.  WHAT  IS  EXPECTED 

•  DT  vs.  OT 

+  Blurry  Demarcation/Combined  T&E 

100# 

•  REQUIREMENTS  “CREEP” 

+  Technology  insertion/Spiral  Development 

•  PERFORMANCE  BASED  SPECIFICATION 
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FACTORS  AFFECTING  T&E 


•  ADVANCING  TECHNOLOGY 

+  Visual  Conformation  of  Target 
+  Higher  resolution  Sensors  (Radar  and  EO) 

+  Laser  Designation 
+  Real  Time  Imagery 

TESTERS  DRIVEN  TO  DEVELOP  TESTS  AND 
PROCEDURES  TO  HANDLE  TECHNOLOGICAL 
DEVELOPMENTS 


•  VEHICLE  INTEGRATION 

+  Treat  System  Under  Test  (SUT)  as  complete  system:  Front  end  -  user 
+  The  aircraft/platform  isn’t  the  lab  (Pay  me  now  or  really  pay  me  later) 
+  Pilot  to  Vehicle  Interfaces 
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DEVELOPMENT  TEST  AND  EVALUATION 


•  CONSTRAINTS 

+  LAB  vs.  Hangar 

+  Location  of  Equipment:  Optical  Bench  vs.  who  knows  where 
+  Variability  of  EO  Sensors 

*  FOV’s,  Apertures,  Scan  Patterns,  Lasers,  etc. 

+  Test  Equipment  is  never  cheap  or  easy  to  maintain 

+  $$,  Politics,  Acquisition  process,  Sponsors,  Time,  blah  blah  blah 
...  Time  to  shoot  engineers  and  get  on  with  the  project... 

•  FLIGHT  vs.  GROUND 

+  Important  to  exercise  SUT  under  loads  (see  M&S  later!) 

*  Hard  to  impossible  to  simulate  A/C  vibration  and  acoustics 
+  Can’t  request  weather  and  environmental  conditions 

+  Human  in  the  loop 
+  Sophisticated  Target  Boards 

TESTER'S  BIGGEST  NO-NO 

Unrealistic  (stupid)  or  Unsafe  TEST 

^  i  “  NAVAL  AVlATlDl 


EO  T&E  EQUIPMENT 
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MODELING  AND  SIMULATION 


•  Move  towards  modeling  based  acquisition 

•  Integral  part  of  T&E  process 

+  Should  not  replace  flight  test 
*  Reduce  and  refine  flight  tests 

•  Pro’s  and  Con’s  for  DT 

•  Sometimes  not  as  cheap  as  presented 

+  Cost  to  develop,  maintain  and  upgrade 
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MODELING  AND  SIMULATION 


ADVANTAGES 

DISADVATAGES 

Allows  1000’s  of  data  runs 

Usually  not  stochastic  (Random  inputs) 

Early  diagnostic  tool  for  design  decisions 

Expensive  to  develop 

Ability  to  test  edge  of  envelope 

Only  as  good  as  data  in 

Cheaper  than  Flight  Test 

Can’t  replicate  all  variables  of  platform 

Provides  “What  if  s?” 

Verification,  Validation  &  Accreditation 
(VV&A) 

Can  allow  inclusion  of  other  sensors  to 
test  integration 

Accuracy/fidelity  cost  and  time  driven 

Provides  input  for  Fleet  battlefield 
Experimentation — Allows  insight  into  the 
“big  picture”  overview  for  operational 
implementation 

Constant  upgrades/maintenance 

MODELING  AND  SIMULATION 


HARDWARE  IN  THE  LOOP 

•  DIGITAL  INJECTION 

+  Repeatable  High  Clutter  Environments 
+  Edge  of  Envelope  Excursions 
+  GIGO 
+  Not  end-end 

+  SUT  must  be  duped  into  flight  mode  (AoA,  INS,  Alt, 
Airspeed,  etc) 

+  Access  points  not  always  accessible 
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SCENE  PROJECTION 


MODELING  AND  SIMULATION 


HARDWARE  IN  THE  LOOP 
•  SCENE  PROJECTION 

+  Project  actual  EO  signals  directly  into  optics 
+  Assume  digital  model  can  drive  projection  equipment 
+  Quick  update  rates  over  wide  dynamic  ranges 
+  Collimated  images  into  a  wide  range  of  FOV’s 
+  Single  or  Multiple  sensors 

*  Staring  or  slewing 

*  More  than  one  aperture 
+  Expensive  to  build/develop 
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MODELING  AND  SIMULATION 

BEST  APPROACH 


•  Combination  of  Digital  Injection,  Scene  Projection  then 
Flight  Test 

•  No  one  “Silver  Bullet” 

•  Utilize  all  tools  in  the  inventory 

•  Limits  regression  testing 

•  Always  use  all  of  your  available  “Tools” 


MODELING  AND  SIMULATION 


VERIFICATION,  VALIDATION  AND  ACCREDIDATION 

•  Convince  T&E  Engineer  models  are  accurate  and 
representative 

+  Must  answer  more  questions  than  it  raises 

•  Only  as  good  as  data  in 

+  Sometimes  too  expensive  to  collect  data,  and  pursue  VV&A 
(Spend  $20M  to  get  the  $100  answer) 

•  Who  funds  the  effort 

•  Budget  Time  and  $$  into  program  for  “tweaks  and  upgrades” 

+  Collect  real  data  to  verify  model  (within  error  bars) 

•  Get  OT  buy  in-  They  need  assurance  that  model  reflects  real 
world 

+  No  “build  it  and  they  will  come” 


FUTURE  TRENDS 


•  Real-Time  Tactical  Imagery 

•  Active  vs.  Passive  Imaging 

•  Multiple  Sensor  Fusion 

•  Information  Dissemination 

+  NCW 
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QUESTIONS/COMMENTS 


POC: 

Raymond  F.  Beach 

NAVAIR  Associate  Fellow/EO-RADAR  Branch  Head 
NAVAL  AIR  WARFARE  CENTER 

Mission  Systems  EvaluationDivision  5. 1.2. 7  BLDG  114  Room  209 
22147  Sears  Road  Unit  4 
Patuxent  River,  MD  20670 


naval  AVIATION  technologies 


Phone:  301-342-6518 
raymond.beach@navy.mil 
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Optimizing  Risk  Management 

Life  at  Level  3 


Level  3  Risk  Practices 


Prepare  for  Risk 
Management 


RSKM  SGI 


Identify  and 

RSKM  SG2 

r 

Analyze  Risks 

Identify  Project 
Risks 


PP  SP2.2 


Establish  Budget 
and  Schedule 


PPSP2.1 


Mitigate 

Risks 


PMC  SP13 


RSKM  GP3.2 


Monitor  Project 
Risks 


Collect  Improvement 
Information 


RSKM  SG3 
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Optimizing  Risk  Management 

Life  at  Level  3 


Establish  the  Budget  &  Schedule 

For  each  risk,  multiply  its  impact  times  its 
probability,  to  get  it’s  exposure 

Add  them  all  together  to  estimate  a  buffer  or 
reserve 

►  Do  this  for  effort 

►  Do  this  for  duration 

►  Do  this  for  costs 

You  could  be  falling  into  a  trap! 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


B  =  SUM(  Fj  *  Pi ) 

Where: 

r,  =  impact  of  risk(i),  i  =  1  to  n 
p,  =  probability  of  risk®,  between  0  and  1.0 
B  =  total  risk  buffer  estimated  for  any  dimension  of  impact: 
cost 
duration 
effort,  etc. 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


The  Problem 


Based  on  B  =  SUM(  r,  *  p, ) 


HALF  of  your 
projects  will  be  late! 
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Would  you  start  a  project  knowing  there  was 
a  50  percent  chance  it  will  be  late  based  on 
risks  alone? 

Would  your  customers  accept  having  50%  of 
their  projects  being  late? 
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Why  the  Problem? 

The  random  variable  to  reflect  the  risk 
outcomes  has  a  distribution  that  looks  like  this: 


The  buffer  to  protect  the  project  from  these  risks,  B,  is  commonly 
set  to  the  expected  value  for  the  sum  of  manifested  risks,  Rbar. 


But,  50%  of  the  outcomes  will  be  greater  than  B. 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


Central  Limit  Theorem  to  the  Rescue 


The  Central  Limit  Theorem  of  statistics 
says  that  the  sums  of  random  variables 
tend  to  become  approximately  normal,  i.e. 
they  follow  a  Gaussian  Curve. 
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Optimizing  Risk  Management 
Avoiding  the  Trap 


How  big  should  buffers  be? 

Applying  this  distribution  to  real 
projects  with  duration  D  means  that 
half  the  projects  will  experience 
outcomes  from  manifested  risks 
that  will  use  up  the  buffer 
and  put  them  over  the 
common  estimate,  E. 

D+R 


E  =  D+R. 


Start 


D 


We  can  estimate  the  standard  deviation  from  the  binomial 


nature  of  our  risks:  a  =  SQRT(SUM(  r,  *  p,  )/n). 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


Choosing  the  Buffer 


And  use  our  estimate  to  calculate  a 
new  buffer,  B' ,  and  new  Estimate, 

E'=  D  +  B'  =  D  +  Rbar  +  (z  *  a) 

‘z’  is  chosen  to  decide  what 
percentage  of  projects 
should  be  expected,  based 
only  on  risk,  to  go  over  their 
estimates. 


Start 


D+R 


v - v - ' 

z  *  a 
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Choose  your  ‘z’ 


Select  a  standard  value  for  ‘z’ 
to  use  to  calculate  risk  buffers. 

z  =  0 

will  give  you  the  protection  you 
have  now. 

z  =  2.0 

will  give  you  risk  buffers  to 
protect  97.72%  of  projects 

(or  phases  of  projects). 

Only  2.28%  would  be  expected 


to  exceed  their  buffer. 


Optimizing  Risk  Management 
Avoiding  the  Trap 


z 

%  expected  to 
be  over 

0.0 

50.00% 

0.5 

30.85% 

1.0 

15.87% 

1.5 

6.68% 

2.0 

2.28% 

2.5 

0.62% 

3.0 

0.13% 

3.5 

0.02% 
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Optimizing  Risk  Management 
Avoiding  the  Trap 


Using  the  New  Buffer 


The  duration  of  projects  will  not  increase. 

The  costs  of  projects  will  not  increase. 

Customers  will  not  pay  more  or  wait  longer  for 
projects. 


Because... 

Not  all  of  risk  buffers  will  be  consumed!  Some  will. 
But,  many  won’t. 

Customers  will  actually  find  that  you  are  early  and 
under-budget  (or  late  and  over-budget)  according 
to  the  ‘z’  factor  that  you  chose. 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


Benefits 


You  will  not  over-promise. 
You  will  not  under-deliver. 


You  won’t  have  to  charge  more. 

You  will  just  have  to  apologize  less, 
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Optimizing  Risk  Management 
Improving  Risk  Management 


How  Do  We  Improve  Risk  Management? 

□  Learn  more  about  risks: 

►  estimate  better 

►  plan  better 

a  Increase  what  we  remember  about 
problems  and  risks 

a  Leverage  lessons  learned 

a  Do  all  this  systematically 
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Optimizing  Risk  Management 

Improving  Risk  Management 


Level  3  Risk  Practices 


Prepare  for  Risk 
Management 


RSKM  SGI 


Identify  and 
Analyze  Risks 


RSKM  SG2 


Identify  Project 
Risks 


PP  SP2.2 


Establish  Budget 
and  Schedule 


SP13 


■ 


. 


PP  SP2.1 


Mitigate 

Risks 


RSKM  GP3.2 


Monitor  Project 
Risks 


Collect  Improvement 
Information 


RSKM  SG3 
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Do  this,  do  that.  Do  this, 
do  that.  Do  this,  do  that. 
Do  this,  do  that.And  then 
do  this.  Stop.  Do  this,  do 
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Optimizing  Risk  Management 

Risk  Management  System 


Elements  of  a  System 

People:  trained  and  assigned  with 
responsibilities  to  operate  and  maintain  the 
system 

Technology:  tools  make  it  easier  to 
operate  the  system.  If  people  find  activities 
too  difficult  they  will  not  be  done  well 

Processes:  defined  and  available  to  guide 
activities  and  contribute  to  greater 
effectiveness,  efficiency,  and  quality 
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RMS  Components 

People,  training,  support,  responsibilities, 
stakeholders,  monitoring  and  supervision  of 
risk  management 


□  Database  of  problems  and  risks  with  historical 
occurrences  and  supporting  information 


□  Mechanism  to  collect  data  on  problems,  risks, 
and  lessons  learned  on  the  success  and  costs 
of  mitigations  and  contingency  plans 

□  Processes,  policies  and  plans  defined  to  guide 
and  give  consistency  to  risk  management 
activities 
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Learned 
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Risk  Management  System 


Projects  Have  Problems 


f 

\ 

Projects 

V 

/ 

Updated  Impacts, 
Probabilities, 
Mitigations, 
Contingencies 


/ 

\ 

Problems 

V 

J 

New 

Problems 
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Optimizing  Risk  Management 

Risk  Management  System 


Learning  About  Risks 


□  Risk  parameters  and  attributes 

□  Actual  risk  impacts 


□  Actual  risk  historical  probabilities 
a  Mitigation  plan  successes 

□  Contingency  plan  successes 


'V\ 
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Risk  Management  System 


Risk  Management  System 


New 

Risks 


Potential  Risks 


Risk 

Database 


Risl^ccurrences, 

and  Impacts 


Lessons 

Learned 


Project 

Planning 


Identified 

Risks 


Project  Execution 
&  Monitoring 


Problems  & 
Successes 
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Optimizing  Risk  Management 

Risk  Management  System 


Risk  Management  System  Goals 


□  Improve  accuracy  of  risk  parameters 
and  contingency  buffers 

□  Improve  identification  of  risks 

□  Improve  mitigation  of  risks  and 
effectiveness  of  contingency  plans 

□  Reduce  the  impact  of  risks  on  project 
objectives 
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Optimizing  Risk  Management 

Risk  Management  System 


Goal  Confirming  Questions 


a  How  stable  are  risk  parameters? 

a  How  accurate  are  risk  estimates  of 
probability  and  impact? 

a  How  often  are  projects  surprised  by  new 
risks  and  problems? 

□  How  effective  are  mitigation  and 
contingency  plans? 
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Optimizing  Risk  Management 

Risk  Management  System 


Goal  Revealing  Measures 


a  Risk  impact  estimates  vs.  actuals 
a  Risk  probability  estimates  vs.  actuals 
a  Frequency  of  new  issues  and  problems 
a  Mitigation  plan  estimates  vs.  actuals 
a  Contingency  plan  estimates  vs.  actuals 
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Optimizing  Risk  Management 

Risk  Management  System 


Using  the  System  at  Level  3 

RSKM  SGI 


Identify  and 
Analyze  Risks 


RSKM  SG2 


Identify  Project 
Risks 


PP  SP2.2 


Establish  Budget 
and  Schedule 

w 

PMC  SP13 


PP  SP2.1 


— * 

Mitigate 

Risks 

Monitor  Project 
Risks 


GP3.2 

Collect  Improvement 
Information 

RSKM  SG3 
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Use  in  Higher  Levels 


Level  4  &  5  Risk  Practices 


RSKM  SGI 


Prepare  for  Risk 
Management 


Identify  and 
Analyze  Risks 


RSKM  GG5 


Institutionalize  an 
Optimized 
Process 


Establish  Budget 
and  Schedule 


RSKM  SG3 


Institutionalize  a 
Quantitatively 
Managed  Process 


Collect  Improvement 
Information 
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Optimizing  Risk  Management 
Use  in  Higher  Levels 


Correct  Root  Causes  of  Problems 


Pareto  Chart 


100% 

80% 

60% 

40% 

20% 

0% 


MGMT.  DEV.  CM  VER.  VAL. 
QMS  Process  Area 
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Optimizing  Risk  Management 

Use  in  Higher  Levels 


Quantitatively  Managed  Process 
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Figure  3 


Individual  Control  Chart 


■UCL  18.12 


V  12.80 


■LCL  7.48 
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4  6  8 

Test  Number 


Uspec  =  19 


Lspec  =  7 
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Optimizing  Risk  Management 

Use  in  Higher  Levels 


Ensure  Continual  Process  Improvement 


Figure  2 


Moving  Range  Control  Chart 
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Individual  Control  Chart 
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Summary 


□  Risk  estimating  can  be  improved  by 
avoiding  a  common  statistical  trap 

□  Risk  Management  can  be  improved  even 
at  Level  3 

□  It  takes  a  system  to  optimize  Risk 
Management 


Hal 


Optimizing  Risk  Management 

Going  Forward 


Conclusion 


a  It  takes  a  system  to  optimize  anything 
a  A  system  where  that  optimization  is  its  AIM 

Where  can  we  apply  such  a  system? 

□  Project  estimating 

□  Requirements  Management 
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Optimizing  Risk  Management 

Going  Forward 


Adding  to  your  System 


OSSP 

Process 

System 
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Optimizing  Risk  Management 

Thank  You! 
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Optimizing  Risk  Management 

Thank  You! 
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Optimizing  Risk  Management 

Life  at  Level  3 


Level  3  Risk  Practices 


Prepare  for  Risk 
Management 


RSKM  SGI 


Identify  and 

RSKM  SG2 

r 

Analyze  Risks 

Identify  Project 
Risks 


PP  SP2.2 


Establish  Budget 
and  Schedule 


PPSP2.1 


Mitigate 

Risks 


PMC  SP13 


RSKM  GP3.2 


Monitor  Project 
Risks 


Collect  Improvement 
Information 


RSKM  SG3 
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Optimizing  Risk  Management 

Life  at  Level  3 


Establish  the  Budget  &  Schedule 

For  each  risk,  multiply  its  impact  times  its 
probability,  to  get  it’s  exposure 

Add  them  all  together  to  estimate  a  buffer  or 
reserve 

►  Do  this  for  effort 

►  Do  this  for  duration 

►  Do  this  for  costs 

You  could  be  falling  into  a  trap! 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


B  =  SUM(  Fj  *  Pi ) 

Where: 

r,  =  impact  of  risk(i),  i  =  1  to  n 
p,  =  probability  of  risk®,  between  0  and  1.0 
B  =  total  risk  buffer  estimated  for  any  dimension  of  impact: 
cost 
duration 
effort,  etc. 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


The  Problem 


Based  on  B  =  SUM(  r,  *  p, ) 


HALF  of  your 
projects  will  be  late! 
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Would  you  start  a  project  knowing  there  was 
a  50  percent  chance  it  will  be  late  based  on 
risks  alone? 

Would  your  customers  accept  having  50%  of 
their  projects  being  late? 
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Why  the  Problem? 

The  random  variable  to  reflect  the  risk 
outcomes  has  a  distribution  that  looks  like  this: 


The  buffer  to  protect  the  project  from  these  risks,  B,  is  commonly 
set  to  the  expected  value  for  the  sum  of  manifested  risks,  Rbar. 


But,  50%  of  the  outcomes  will  be  greater  than  B. 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


Central  Limit  Theorem  to  the  Rescue 


The  Central  Limit  Theorem  of  statistics 
says  that  the  sums  of  random  variables 
tend  to  become  approximately  normal,  i.e. 
they  follow  a  Gaussian  Curve. 


©  2006  Natural  SPI 


Hal 


Optimizing  Risk  Management 
Avoiding  the  Trap 


How  big  should  buffers  be? 

Applying  this  distribution  to  real 
projects  with  duration  D  means  that 
half  the  projects  will  experience 
outcomes  from  manifested  risks 
that  will  use  up  the  buffer 
and  put  them  over  the 
common  estimate,  E. 

D+R 


E  =  D+R. 


Start 


D 


We  can  estimate  the  standard  deviation  from  the  binomial 


nature  of  our  risks:  a  =  SQRT(SUM(  r,  *  p,  )/n). 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


Choosing  the  Buffer 


And  use  our  estimate  to  calculate  a 
new  buffer,  B' ,  and  new  Estimate, 

E'=  D  +  B'  =  D  +  Rbar  +  (z  *  a) 

‘z’  is  chosen  to  decide  what 
percentage  of  projects 
should  be  expected,  based 
only  on  risk,  to  go  over  their 
estimates. 


Start 


D+R 


v - v - ' 

z  *  a 
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Choose  your  ‘z’ 


Select  a  standard  value  for  ‘z’ 
to  use  to  calculate  risk  buffers. 

z  =  0 

will  give  you  the  protection  you 
have  now. 

z  =  2.0 

will  give  you  risk  buffers  to 
protect  97.72%  of  projects 

(or  phases  of  projects). 

Only  2.28%  would  be  expected 


to  exceed  their  buffer. 


Optimizing  Risk  Management 
Avoiding  the  Trap 


z 

%  expected  to 
be  over 

0.0 

50.00% 

0.5 

30.85% 

1.0 

15.87% 

1.5 

6.68% 

2.0 

2.28% 

2.5 

0.62% 

3.0 

0.13% 

3.5 

0.02% 
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Optimizing  Risk  Management 
Avoiding  the  Trap 


Using  the  New  Buffer 


The  duration  of  projects  will  not  increase. 

The  costs  of  projects  will  not  increase. 

Customers  will  not  pay  more  or  wait  longer  for 
projects. 


Because... 

Not  all  of  risk  buffers  will  be  consumed!  Some  will. 
But,  many  won’t. 

Customers  will  actually  find  that  you  are  early  and 
under-budget  (or  late  and  over-budget)  according 
to  the  ‘z’  factor  that  you  chose. 
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Optimizing  Risk  Management 

Avoiding  the  Trap 


Benefits 


You  will  not  over-promise. 
You  will  not  under-deliver. 


You  won’t  have  to  charge  more. 

You  will  just  have  to  apologize  less, 
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Optimizing  Risk  Management 
Improving  Risk  Management 


How  Do  We  Improve  Risk  Management? 

□  Learn  more  about  risks: 

►  estimate  better 

►  plan  better 

a  Increase  what  we  remember  about 
problems  and  risks 

a  Leverage  lessons  learned 

a  Do  all  this  systematically 
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Optimizing  Risk  Management 

Improving  Risk  Management 


Level  3  Risk  Practices 


Prepare  for  Risk 
Management 


RSKM  SGI 


Identify  and 
Analyze  Risks 


RSKM  SG2 


Identify  Project 
Risks 


PP  SP2.2 


Establish  Budget 
and  Schedule 


SP13 


■ 


. 


PP  SP2.1 


Mitigate 

Risks 


RSKM  GP3.2 


Monitor  Project 
Risks 


Collect  Improvement 
Information 


RSKM  SG3 
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Do  this,  do  that.  Do  this, 
do  that.  Do  this,  do  that. 
Do  this,  do  that.And  then 
do  this.  Stop.  Do  this,  do 
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Optimizing  Risk  Management 

Risk  Management  System 


Elements  of  a  System 

People:  trained  and  assigned  with 
responsibilities  to  operate  and  maintain  the 
system 

Technology:  tools  make  it  easier  to 
operate  the  system.  If  people  find  activities 
too  difficult  they  will  not  be  done  well 

Processes:  defined  and  available  to  guide 
activities  and  contribute  to  greater 
effectiveness,  efficiency,  and  quality 
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RMS  Components 

People,  training,  support,  responsibilities, 
stakeholders,  monitoring  and  supervision  of 
risk  management 


□  Database  of  problems  and  risks  with  historical 
occurrences  and  supporting  information 


□  Mechanism  to  collect  data  on  problems,  risks, 
and  lessons  learned  on  the  success  and  costs 
of  mitigations  and  contingency  plans 

□  Processes,  policies  and  plans  defined  to  guide 
and  give  consistency  to  risk  management 
activities 
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Risk  Management  System 


Projects  Have  Problems 


f 

\ 

Projects 

V 

/ 

Updated  Impacts, 
Probabilities, 
Mitigations, 
Contingencies 
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\ 

Problems 
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New 

Problems 
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Optimizing  Risk  Management 

Risk  Management  System 


Learning  About  Risks 


□  Risk  parameters  and  attributes 

□  Actual  risk  impacts 


□  Actual  risk  historical  probabilities 
a  Mitigation  plan  successes 

□  Contingency  plan  successes 


'V\ 
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Analysis 
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Risk  Management  System 


Risk  Management  System 


New 

Risks 


Potential  Risks 


Risk 

Database 


Risl^ccurrences, 

and  Impacts 


Lessons 

Learned 


Project 

Planning 


Identified 

Risks 


Project  Execution 
&  Monitoring 


Problems  & 
Successes 
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Optimizing  Risk  Management 

Risk  Management  System 


Risk  Management  System  Goals 


□  Improve  accuracy  of  risk  parameters 
and  contingency  buffers 

□  Improve  identification  of  risks 

□  Improve  mitigation  of  risks  and 
effectiveness  of  contingency  plans 

□  Reduce  the  impact  of  risks  on  project 
objectives 
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Optimizing  Risk  Management 

Risk  Management  System 


Goal  Confirming  Questions 


a  How  stable  are  risk  parameters? 

a  How  accurate  are  risk  estimates  of 
probability  and  impact? 

a  How  often  are  projects  surprised  by  new 
risks  and  problems? 

□  How  effective  are  mitigation  and 
contingency  plans? 
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Optimizing  Risk  Management 

Risk  Management  System 


Goal  Revealing  Measures 


a  Risk  impact  estimates  vs.  actuals 
a  Risk  probability  estimates  vs.  actuals 
a  Frequency  of  new  issues  and  problems 
a  Mitigation  plan  estimates  vs.  actuals 
a  Contingency  plan  estimates  vs.  actuals 
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Optimizing  Risk  Management 

Risk  Management  System 


Using  the  System  at  Level  3 

RSKM  SGI 


Identify  and 
Analyze  Risks 


RSKM  SG2 


Identify  Project 
Risks 


PP  SP2.2 


Establish  Budget 
and  Schedule 

w 

PMC  SP13 


PP  SP2.1 


— * 

Mitigate 

Risks 

Monitor  Project 
Risks 


GP3.2 

Collect  Improvement 
Information 

RSKM  SG3 
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Use  in  Higher  Levels 


Level  4  &  5  Risk  Practices 


RSKM  SGI 


Prepare  for  Risk 
Management 


Identify  and 
Analyze  Risks 


RSKM  GG5 


Institutionalize  an 
Optimized 
Process 


Establish  Budget 
and  Schedule 


RSKM  SG3 


Institutionalize  a 
Quantitatively 
Managed  Process 


Collect  Improvement 
Information 
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Optimizing  Risk  Management 
Use  in  Higher  Levels 


Correct  Root  Causes  of  Problems 


Pareto  Chart 
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Stabilize  Subprocess  Performance 
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Optimizing  Risk  Management 

Use  in  Higher  Levels 


Quantitatively  Managed  Process 
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Figure  3 


Individual  Control  Chart 


■UCL  18.12 


V  12.80 


■LCL  7.48 


-i - - - 1 - 1 - r- 

4  6  8 

Test  Number 


Uspec  =  19 


Lspec  =  7 
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Optimizing  Risk  Management 

Use  in  Higher  Levels 


Ensure  Continual  Process  Improvement 


Figure  2 


Moving  Range  Control  Chart 
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Summary 


□  Risk  estimating  can  be  improved  by 
avoiding  a  common  statistical  trap 

□  Risk  Management  can  be  improved  even 
at  Level  3 

□  It  takes  a  system  to  optimize  Risk 
Management 


Hal 


Optimizing  Risk  Management 

Going  Forward 


Conclusion 


a  It  takes  a  system  to  optimize  anything 
a  A  system  where  that  optimization  is  its  AIM 

Where  can  we  apply  such  a  system? 

□  Project  estimating 

□  Requirements  Management 
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Optimizing  Risk  Management 

Going  Forward 


Adding  to  your  System 


OSSP 

Process 

System 
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Optimizing  Risk  Management 
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Booz  Allen  Hamilton 


Agenda 


►  Background  of  Logistics  Readiness  Level  (LRL)  concept 

►  LRL  Defined 

►  LRL  Tool 

►  Prototype  Example 

►  Benefits 

►  Questions 
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Background 


►  LRL  concept  evolved  from  the  DOD  5000.2  mandated  Technology  Readiness 
Levels  (TRL)  assessment  process 

-  TRLs  provide: 

•Evaluation  of  critical  technology  maturity 
•Maturation  plan  (as  needed) 

•Best  practices/guidelines  for  each  Milestone 
■MS  B  target  is  TRL  =  6 
■MS  C  target  is  TRL  =  7 
■MS  C  preferred  is  TRL  =  8 

►  Logistics  benchmark  system  for  technology  insertion  was  desirable 

-  Aid  in  understanding  what  sustainment  is  required  at  different  time  phases 

-  Technology  can  be  mature  with  immature  logistics  support 

LRL  is  a  new  concept  intended  to  consider 
sustainment  issues  for  tech  insertion 

projects 
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LRL  Definition 

►  LRL  intent: 

-  Provide  a  methodology  and  benchmarks  for  assessing  Logistic  Element  Readiness  for 
technology 

-  Provide  a  management  tool  to  forecast  logistics  workload,  manpower  requirements, 
identify  gaps,  etc. 

-  LRL's  evaluated  for  6  project  phases: 

•Lab  Test/  R&D 

•Project  Definition  (Fleet  Need/ metrics/ BCA/ Decision  to  proceed) 

•Project  Development /I  mplementati on  (Finalized  analysis,  change  recommended,  ECP 
development.  Class  1 1  change  development,  RAMEC,  LECP,  other) 

•Engineering  Validation 

•Fleet  Verification 

•Fleet  Use 


Engineers  and  Logisticians  need  clear  definition 
of  what  is  required  for  sustainment  at  each 

phase  of  a  project 
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LRL  Definition 

►  Evaluation  criteria  established  for  each  phase 

-  Determined  the  benchmark  of  required  tasks  appropriate  for  each  logistics  element  at 
each  time  phase 

-  Not  all  elements  require  same  level  or  effort  in  the  same  time  phase 

►  Answers  question  of  “What  tasks  must  be  complete  at  each  project  phase  for 
each  logistics  element?" 

-  Training  example: 


Phase 

Lab 
Test  / 
R&D 
phase 

Project 

Definition 

Project  Development/ 
Implementation 

Engineering 

Validation 

Fleet  Verification 

Fleet  Use 

Training 

Existing 

training 
procedures/ 
curricula  and 
training  plan 

identified  and 
reviewed. 

Impacts  to  Training 
identified 

Training 

curricula  changes 
drafted.  As 
required  changes 

to  Naval  Training 
Systems  Plan 
(NTSP)  drafted. 

Training  curricula 

changes  updated 
post 

validation/verification 
with  changes  as 
necessary.  NTSP 
changes  finalized. 
Changes  submitted 
for  approval. 

Training 

curricula 

updated. 

NTSP 

updated. 

Booz  Allen  Hamilton 
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LRL  Tool _ 

►  LRL  tool  created  in  Microsoft  Excel  (compatible  with  Microsoft  Office  2003) 
includes  instructions  worksheet 

-  Buttons  allow  a  point  and  click  use  of  color  for  evaluation 

•Grey  indicates  the  project  is  not  yet  in  that  project  phase  therefore  benchmarks  are  not 
applicable 

•Blue  indicates  the  benchmark  is  not  applicable  due  to  the  details  of  the  project 
•Red  indicates  the  benchmark  task  is  not  complete 
•Green  indicates  the  benchmark  task  is  complete 

-  Detailed  evaluation  requires  familiarity  with  the  project  and  tasks  completed 
to  date 

►  Numeric  LRL  score  evaluated  for  each  project  phase 

-  LRL  =  0  Unsupported  (zero%  required  tasks  completed) 

-  LRL  =  1  Poorly  Supported  (1-50%  of  required  tasks  completed) 

-  LRL  =  2  Moderately  Supported  (51-70%  or  required  tasks  completed) 

-  LRL  =  3  Nearly  Supported  (71-99%  of  required  tasks  completed) 

-  LRL  =  4  Fully  Supported  (100%  of  required  tasks  completed) 

►  Numeric  LRL  scale  provides  a  reference  framework  for  project  comparison 

An  LRL  of  4  is  the  goal  for  all  phases 


Booz  Allen  Hamilton 


5 


LRL  Tool  Screenshot  -  demo  available 
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2  Project  Title: 


Prototype  2  SBRFFT  -  Austere 
Bonding  jnd  Containment  Kits 


3  Project  Descriptioi 


V  jlid  jte  utility  of  bonding 
containment  hit 


4  Project  l^ple^ental 


FST  Update  Technical  Publications 


Fleet  Verification 


5  Project  Phase: 


Date  Efalaated: 


Not  Complete 
Complete 

Not  applicable  due  to  project  specifics 
Not  applicable  due  to  project  phase 


Color  current  cell  red. 


Color  current  cell  green. 


Color  current  cell  blue. 


Color  current  cell  grey. 


Recalculate 


I  Lnqirtior  Rd-adir.d-rr  Ld-'.'d-l  0  -  Unruppnrtd-d.  Ld-'.'d-l  1  -  Poorly  Suppnrtd-d.  Ld-'.'d-l  2  -  Mndd-ratd-ly  Suppnrtd-d.  Ld-'.'d-l  3  -  Hd-arly  S'jppnrtd-d.  Ld-'.'d-H  -  Fully  S'jppnrtd-d. 


Lab  TestfRtD  phase 


Project  Defiiitioi 
CFI**t 

H»tJI>»tricrlBCAfD»cirii> 


Logistics 
Readiness  Level 


Design 

Interface 


Rd-uid-u  -and  idd-ntifyriqnifiaant  Juiii 
■  -a-c*-  impaatr  nf  prn jc--:t  tn 

d-xirtinqjyrtd-m  nr  platf  nrm  (d-K. 

■-ail-a t Id-  pnud-r,  ud-iqht  cnnrtraintr, 
d-tc.JCrd-atd-POAMtn  rd-rnlud-  any  dd-riqn 
intd-rfacd-  irrud-r. 
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Exirtinq  Rt-li-nfc-ilit  j-  «J 
H«i»t<i>«lility  (RAM)md-trior 
rd-'.'id-ud-d.  Initial  imprnud-md-nt 
prd-di-itinnr -Jd-td-rmind-d.  Di-xi^k 
iktkrf-ic#  Ltj'Jl-  rd-rnlutinn  in  unrk. 


Project  Development 
flapleBeitatiok 
(Fikiliikd  fkkljjir.  ckkki* 

ECP  Jkkkl^.kkt, 


Fnr  r.d-u  dd-riqnr,  Ro-li-nk-ilit t  Ckittri-I 
H«ik(»«k«  (RCH)  -and  F-nilar* 
Hadu  «J  Effkctr  Akkljrir 
(FHEA)  -enmpld-td-d  tn  identify  failurd- 
mndd-r,  F-ailurd-  frd-qud-nay,  d- F F d- h  nn 
pd-rfnrman-id-,  -and  criticality.  Fnr 
mndificatinnrtnd-Kirtinaddjian.RCMand 


Eagiieeriig  Validation 


Fid-rultr  of  RCH  -and  FHEA  urd-dtn 
dd-'.'d-lnp  nr  mndify  d-xirtinq  ennditinn 
bar  d-d  and  xchd-duld-  bar  d-d  maintd-naned- 
tarkr.  Rd-rultr  nf  RCM  and  FMECfl  aLrn 
urd-d  tn  updatd-  thd-  Critical  ltd-mr  lirt  or 
appli-zabld-.  Td-chnical  data  updatd-r 
drafted  and  ualidatd-d. 


Fleet  Verification 


T r-ckki-c-al  ipditrr  (ruch-ir 
Maintd-naned-  Rd-quird-md-nt  Card 
ahanqda-)  vd-riFid-d. 


Nf  A 


Tk-cLkical  updatd-r 
-:nmpld-td-d  and  auailabld-. 


PriJacI  DrauikqlUcL 
D-nt-n  Packaqi-  d-ld-md-r.tr 
cnmpld-td-d  includinq  or 
appli-zabld-: 

Spd-crftd-ch 

manualrfd-nqind-d-rinq  drauin-qr 

SaHd-ty  Rd-quird-md-r.tr 

Prd-rd-ruatinn  and  packaqinq 

rd-quird-md-ntr 

Td-rt  rd-quir L-mL-r.tr  data 

Prd-ud-ntatiud- 


H  ^  ►  n\  LRL  Instructions  /  Prototype  4  HFFT  Rev  2  /  Protype  4  HFFT  Rev  1  /  Prototype  4  HFFT  /  Prototype  3  CP2FFT  \ Prototype  2  SBRFFT/  Prototype  1  CFFT 


E 

3ooz 

Allen 

Hamilton 
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Prototype  Example 

►  Project:  Update  and  create  a  J  oint  General  Series  Hydraulic  Publication  (NA- 
01- 1A) 

►  Project  I  mplementation:  New  Manual  Release 

►  Project  Phase:  Project  Development 


Booz  Allen  Hamilton 
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Prototype  Example-  First  Assessment  LRL  =  0 

First  Assessment  3/24/06,  LRL  =  0,  Issues  with  Technical  Data  and  Training 


1 _ A _ | 

B 

1 

2 

Project  Title: 

Proto. 4  HFFT-Update  -IT  Manual 
and  create  Joint  Publication 

> 

Project  Description 

Update  GENERAL  Series  Technical 
Publication 

4 

Project  Implements 

New  Manual  Release  j 

5 

Project  Phase: 

Project  Development 

t 

Date  Evaluated: 

3/24/2006 

I 


Not  Complete 
Complete 

Not  applicable  due  to  project  specifics 
Not  applicable  due  to  project  phase 


Lnqixfcicx  F: q  -d i r. l-j-j  Li-' 


L- 1  -  I J rj u p p □  r ■: c- -d .  Livil  1  -  F'nnrly  Suppnrfcc-d.  Li  '.'i  l  1  -  Mn  di  r  ihL-ly  S'jf-f  nrti  J.  Liuil  3  -  Hi-irly  Support 


Color  current  cell  red. 

Color  current  cell  green. 
Color  current  cell  blue. 

Color  current  cell  grey. 

Recalculate 

»d.  Li-”i-l  4  -  Fully  Suppnrfcc-d. 

Eagiaeeriag  Validation 

NIA 

Rbrultrnf  RCH  ondFHEA  uridto 
divilap  or  modify  ixirtinq  ennditinn 
b- -3X l- -J  -□  r. -J x ■: hi l- -J -j I l-  b- -3X l- -J 
hoxkx.  Rcxultxnf  RCMir.dFMECfl-ilrn 
uxc-d  hn  updabi  bhi  Critical  Ibtnu  I  Lx  b  ox 
■3  p-  f-  I  i  -=  -3  b- 1  ii- .  Tc-chni-zal  daba  updab^x 
drqfbcd  -3 n -d  ■■■qlidabc-d. 

Pr.d.it  Dr.uiMlTtck  D.t. 
P«ck«*t  ii- 1 i- it. l- r. bx  -d  r  q  f  b 
dtvf-lDpminb  in-zludinq  or  applicable-: 

S  p-  c-  -zx  H  b l- h  rr. -3 r. u  a  Lx  / c- r. -q i  r.  l-  c-  r  i r. -q 

S-sfi-bv  Re-quir c- it. l- r. tx 
Prcxc-ruabmnandpackaqinq 

T*xh  r i- -q u i r i- rr. c- r. hx  -d -3 h -3 
PriVinMbivf. 

M  ■□in':  l-  r.  n  -z  l-  i1  M  ■□in':  l-  r.  n  -z  l- 

Dir»cti*>  drafb 
zamplc  bc  and  ■■■qlidabc-d 

Ttckiic«l  P>kli«ti»XDUr» 

-dab-3,  Flap i-d  A -zbinr.  Chanqex  (R  ACx)  -3 r. ■: 
Mar. ual  Char.-qe-  Fie-que-xtx  (MCRr  J 
v-gl  bid  tar  ennbe-nb  and  format 

Pl»  Uf-d.^L. 

snmc  k  tt  J  -u  ac  E  li  =  at-li:- 

H-ai. *»-»-£: *  C»»rt  up-dabe- 
cnmple-be-doxapplicable- 

L«  ,  I  nf  . »»#R*F4 

Aircraft  B-attla-  D  ■  *-  R»p.«i 


Lab  TestlRLD  phase 


Project  Definition 

(H**t 

•  Jf»tricrfBCAfD*c 
*■  tr.ce.dl 


Project  Development 
Jlnplenentation 

{Fi.ali.ed  »alTxir.  ck«4. 

cnnn*>d*J.  ECP 


Fleet  Verification 


Fleet  Use 


Logistics 
Readiness  Le 


NIA 


Design 

Interface 


■  ie-uandide-nbifyxiqnificanb  duiqa 

trfre*  i  IT  I F"  -3  ■:  h_x  13  i  prnjiib  bn 

■binqxyxbe-m  nr  platform  (tx. 
ilable-  F-nui-r.  ue-iqhb  -znnxbrair.bx, 
i  b  :.  j  CrL  obL-  F'OhM  bn  ri-xnl'.';-  -jr.v  di-xi-qr 
inbi-rh-j-: e-  Lrx -j ex . 


ExirbinqRaliakilitr  -mm* 
Hai.tai»kilitr  (RAM}  me-bri-sr 
re-vie-ue-d.  Initial  imprnue-me-nb 
pre-diebinnx  de-be-rmine-d.  Du-iqa 
i>t*rf<»  Lrx  'j  l-  rornlubinn  in  uork. 


Fnr  niudixiqiir,  Ralial-ilitr  Cratrrr  j 
Haiateaoc*  (RCH)  bindFailar* 
Hadax  »d  Effactr  AaaljxLr 
(FHEA)  c ample- 1 e-d  bn  id e-nbify  failure 
mad  ex.  failure-  fre-que-ncy.e-ffe-cb  an 
performance,  and  cribiealiby.  Far 
mndif  icabinnx  bn  e-xixbinq  dexian.  RCM  and 


Ttckaicrl  ipJrtM  (x'jeh  ax 
Mainbe-nan-ze-  Fle-quire-me-nb  Card 
-zhanqe-xj  >.>e-rif ie-d. 


ckiical  updaber 
TiF-le-te-.d  and  available-. 


Technical  Data 


Original  E-quipment  M-inuf  aeburer  Kiqher 
>1  drauin-q,  publicabinn.xpe-cificabmn 
auailable- 

Applicable-Mabe-rialSafe-byDabaShe-e-tx 
(MSDS}  Available 


daft  e-le-me-r.bx  re- vie-ue-d  and 
uire-me-nbx  i-de-nbif ie-d  in-zludinq  ax 
li-zable-: 

^cxHbe-ch  manuaLr/e-nqine-e-rinq 
uinqx 

e-ty  Re-quire-me-nbx 
xe-ruabmn  and  packaqinq 
uire-me-ntx 
t  re- -quire- rr.e-ntx  data 


nbabiu 


PriJact DrauiaqfTack  Data 
Packa-ia  e-le-me-ntx  draft  de-ue-lapme-nb 
inq I u din q  ax  applicable-: 


Pradact  Draui.qfTack  Data 
Packaqa  e-le-me-ntx  updabe-d  in-zludinq 
or  applicable-: 

Spe-axfbe-chmanuaLrfe-nqine-e-rinq 


Sab  e-ty  Re-quir 


Text  re-q uire-me-ntx  dab- 
F" r e- e- r. b a t i e-  M- 
Fie-quire-me-ntx  Car-dx 


a-zkaqin-q  re--quire-me-r.br 


e-r.an-z  e-fMa  ir.be-r.ar.ee- 


Safe-by  Re-quire-me-nbx 
Prexervation  and  pa-ckaqinq 

Texb  re--quire-me-r.tr  data 
Fre-ue-nbabiue- 

Mainbe-nance-HMainbe-nance- 


Pradact Draui.qfTack 
Data  Packaq#  e-le-me-ntx 
cample-te-dincludinqar 
applicable-: 

Spe-cx/te-ch 

manuaLxHe-nqine-e-rinq  drauinqx 

Safe-byRe-quire-me-ntx 

Prexe-ruatian  and  packaqinq 

re--quire-me-r.tr 

Text  re-quire-me-ntx  data 

Pre-ue-nbabiue- 


Tackaical  Di 


tiaa  draft  ir 


Tackaical  Diractiva  draft  ue-rifie-d 
andupdate-d 


Inxbrucbicmx  >.>e-rif ie-d  far  canbe-nb 


Tackaical  Paklicatiaa 
(ar  Pak  Ckaaqu  aia 

RACx  ar  HCRr)  finalixe-d. 


Exirtinq  Haiataaaaca  Rl-a 
i  -J  a-  n  t  i  f  i  a  - J  -a  ri  -J  r  a-  ■■■  i  a-  lj  a-  -J 


Maintenance 

Planning 


■□nd  rc-p 

n-.-erhaull 


raft  Battla  Daaaq 


Haiataaaaca  Plaa  update- drafte-d -a 

acpli:-at-li:- 


Haiataaaaca  Caacapt  update- 
drafbe-daxapplicable- 


raft  Battla  Daaaqa  Rap 


Haiataaaaca  Plaa  update 

aamc-k  tii  d  ir  -ac-c  li  a  at  li: 


Haiataaaaca  Plaa  update 

■samB-lptP-d-arqB-E-li-s-ab-lp 


completed  ox  applicable 


*-  Caacap 

update-  ■: □  m p I e- 1 e- -d  ax 
acclizable 


raft  Battla  Dasaq 


Training 


Traiaiaq  carricala  chanqex 
drafte-d.  Ax  required  chanqex  to  Naval 
Traiaiaq  Sjxtaax  Plaa  (HTSP) 

drafte-d. 


Traiaiaq  carricala  chanqex 
update-d  par t  ualidati an/ue-rificab inn 
uith  -chanqex  ax  necexxary.  HTSP 
chanqex  finalixe-d.  Chanaexxubmitte-d 


HTSP  update-d. 


|  Current  F-x-cilitia^r  revieueda 

|  i-de-ntif ie-d.  When  applicable-,  f  acilibiex 


7Tm|\  LRL  Instructions  /  Prototype  4  HFFT  Rev  2  X  Protype  4  HFFT  Rev  1  \  Prototype  -4  HFFT  /  Prototype  3  CQ2FFT  /  Prototype  2  SERFFT  /_  Prototype  1  CFFT  X 
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Prototype  Example  Second  Assessment  LRL  =  1 

Second  Assessment  5/4/06,  LRL  =1,  Issues  with  Training 


fl  E= 

c 

i 1  F  - 

G 

1 

£ 

Project  Title: 

Proto. 4  HFFT-Updatc  -IT  Manual 
and  create  Joint  Publication 

Not  Complete 

Color  current  cell  red. 

3 

Project  Description 

Update  GENERAL  Series  Technical 
Publication 

Grey  Background  = 

Complete 

Color  current  cell  green. 

4 

Project  iBpleBektat 

New  Manual  Release 

Not  applicable  due  to  project  specifics 

Color  current  cell  blue. 

5 

Project  Phase: 

Project  Development 

Not  applicable  due  to  project  phase 

Color  current  cell  grey. 

f- 

Date  Evaluated: 

Rev  1  5/4/06 

Recalculate 

T 

Loqirticr  LfiU^I  4  -  Unruppnr^J.  Levell-PoorlySupported.  Li-vf-l  £  -  Suppnr^J.  Level  3  -  Nearly  Sup-p-nr ted.  Level-1  -  Fully  Supported. 

$ 

Phase 

Lab  TestJRtD  phase 

Project  Definition 

CH..I 

H»edlB»tricrlBCAHDecirH> 

Project  Devdopaekt 
llapkautatiok 

IF . red  a.alyxir,  ckaaqe 

Ekgikeerikg  Validation 

Fleet  Verification 

Fleet  Use 

■H 

Logistics 

Readikess  Level 

NIA 

0 

1 

NIA 

NIA 

NIA 

10 

Design 

Interface 

Revieu  -and  identif  yxiqnif  ica  r.t  Ju-iqa 
■  attrl-flct  impactr  of  pro ject  to 
exirtinq  xyxtem  nr  platform  it-:-:. 
gv-iilgtlf-  pnui-r,  uf-iqKl:  fniirtrointr, 
etc.)  Create  F'OhM  to  rt/olMi-  qny  dexiqr. 
interface  irjuu. 

Exirtinq  Reli-abililj  »1 
H«»t«>abililr  (RAM)  metrior 
revieued.  Initial  improvement 
predictionr  determined.  DeriqB 

For  neu  dexiqrir,  Ra-liah-ilit  y  Ceatere-I 

H.Jex  a.J  Effectr  A.alyxir 
(FHEA)  completed  to  identify  failure 
modex,  failure  frequency,  effect  on 
performance,  and  criticality .  For 
modif ica tionr  to  exirtinq  dexian.  ROM  and 

Rexultr  of  RCH  and  FHEA  iredto 
develop  or  modify  exirtinq  condition 
bared  andxchedule  bared  maintenance 

tarkr.  Rexultr  of  RCM  and  FMECA  alro 
ured  to  update  the  Critical  Itemr  lir t  ar 
applicable.  Technical  data  updatex 
drafted  and  validated. 

TecL.ical  «pj«t«  (xuchar 
Maintenance  Requirement  Card 
chanqex)  verified. 

TecL.ical  updatex 
completed  and  available. 

11 

12 

Technical  Data 

Or iqinal  Equipment  Manufacturer  hiqher 
leveldrauinq, publication, xpecification 

-3  -3  i  1  -3  b- 1  >:- 

Applicable  Material  Safety  Data  Sheetr 
(MSDS)  Available 

Pr.J«t  Draui.qlTecL  Data 
Pa-ckaqe  elemer.tr  re vieued  and 
requirementr  identif  ied  includinq  ar 
applicable: 

Specr/techmanuaLr/enqineerinq 

drauinqr 

Safety  Requirementr 

Prexervationandpackaqinq 

requirementr 

Text  requirementr  data 

Preventative 

Pr.Jacl  Draui.qfTecb  Data 
Packaqe  elemer.tr  draft  development 
includinqar  applicable: 

Spear/tech  manualrf enqineerinq  drauinqr 
Safety  Requirementr 

Prexervation  and  packaqinq  requirementr 
Text  requirementr  data 

Preventative  MaintenancefMaintenance 
Requirementr  Car -lr 

Pr.Jvct  Draui.qfTacb  Data 
Packaqa  elemer.tr  draft 
development  includinq  ar  applicable: 
Spear/tech  manualrf  enqineerinq 
drauinqr 

Safety  Requirementr 

Prexervation  and  packaqinq 

Text  requirementr  data 

Preventative 

MaintenancefMaintenance 

PraJvct Draui.qfTecL  Data 
Paclav  elemer.tr  updated  includinq 
ar  applicable: 

Spear/techmanuaLr/enqineerinq 

drauinqr 

Safety  Requirementr 

Prexervation  and  packaqinq 
requirementr 

Text  requirementr  data 

Preventative 

Maintenance/Maintenance 

Pr.Jvct Draui.qfTecL 
Data  Package  elemer.tr 
completed  includinq  ar 
applicable: 

Spear/tech 

manualrHenqineerinq  drauinqr 
Safety  Requirementr 
Prexervation  and  packaqinq 
requirementr 

Text  requirementr  data 
Preventative 

13 

Technical  Directive  Methodoloqy 
IdentiHied 

TecL.ical  Directive  draft  in  proeexx 

complete  and  validated 

TecL.ical  Directive  draft  verified 
and  updated 

TecL.ical  Directive 

releared  and  available 

14 

TecL.ical  Pvblicatio.  xaurce  data, 
Rapid  Action  Chanqex  (R ACx),  and  Manual 
ChanqeRequextr(MCRr)draftedper 
format 

TecL.ical  Pvblicatia.xource 

da t a,  Flap-id  Action  Chanqex  (R ACr )  and 
ManualChanqeRequextr(MCRr) 
validated  for  content  and  format 

TecL.ical  PvLIicati.. 

Inr  t  rue  tin  nr  verified  for  content 

TecL.ical  PvLIicati.. 
(ar  Pah-  CkaM«  wia 

RACr  ar  HCRr)  finalized, 
approved  and  available  for  ure 

15 

Maintenance 

Planning 

10 

Exirtinq  Haiataiaict  Plaa 

identified  and  revieued 

Hai.ttiaict  Pla.  update  drafted  ar 
aeelicable 

comeletedaraeelicable 

comeletedaraeelicable 

comeletedaraeelicable 

IT 

Hai.te.a.c#  Ci.cept  update 
completed  ar  applicable 

update  completed  ar 
aeelicable 

(conxumable,  inrpect  and  repair  ar 
necexxarv.  overhaul) 

drafted  ar  applicable 

completed  ar  applicable 

10 

Level  mf  Ha!.te.a.celrepa!r 

Level  mf  Hvi.te.a.cefRepvir 

Level  mf  Hai.te.a.cefRepair 

Level  mf  Hai.te.a.cefRepair 

Level  mf 

Aircraft  Battle  Dasaqe 

Aircraft  Battle-  DaBaqt  Repair 

Aircraft  Battle  Dana-qa-  Repair 

Aircraft  Battle-  D»aqe 

Aircraft  Battle  Daaaqe 

£0 

£1 

Training 

lj*  .>  ^  !■* -i;  .  drafted.  Ar  required  ah anqej  to  Haaal 

H  Trai.i.q  Sjiteu  Pla*  (MTSP) 

Trai.i.q  cvrricvla  chanqex 
updated  part  validation/ verif ication 
uith  chanqex -sx  necexxarv.  HTSP 

Trai.i.q  curricula  updated. 
HTSP  updated. 

££ 

Current  F ■o-cilitiM-  revieued  and  impactr 
identified,  ^fhen  applicable,  facilitiex 

ir  r.s.s.As.A  .nhkhMr.j;, . 

H  ^  ►  h|\  LRL  Instructions  /  Prototype  4  HFFT  Rev  2  \  Pro-type  4  H  FFT  Rev  1  /  Prototype  4  HFFT  /  Prototype  3  CQ2FFT  /  Prototype  2  SBRFFT  / 

Prototype  1  CFFT  / 

Booz  Allen  Hamilton 


9 


Prototype  Example  Third  Assessment  LRL  =  4 


Third  Assessment  5/30/06,  LRL  =  4,  all  benchmarks  achieved 


A  B 

C 

D  1 

E  F  Q 

1 

1 

E 

Project  Title: 

Proto. 4  HFFT-Updatc  -IT  Manual 
and  create  Joint  Publication 

Red  Background  = 

Not  Complete 

Color  current  cell  red. 

4 

5 

Project  Description 

Update  GENERAL  Series  Technical 
Publication 

Green  Background  = 

Complete 

Color  current  cell  green. 

Project  Inplenentat 

New  Manual  Release 

Blue  Background  = 

Not  applicable  due  to  project  specifics 

Color  current  cell  blue. 

Project  Phase: 

Project  Development 

Grey  Background  = 

Not  applicable  due  to  project  phase 

Color  current  cell  grey. 

t 

Date  Evaluated: 

Rev  2  5/30/06 

Recalculate 

T 

$ 

* 

10 

11 

Loqixticx  Readinexx  Level  0  -  Uiirup-p-Dr^  -: 

d.  Level  1  -  Poorly  Supported.  Lf-vi-l  E  -  Mn-df-r'itf-ly  SuFFortf-d.  Level  3  -  Nearly  Supported.  Level  4  -  Fully  Supported.  I 

Phase 

Lab  TestlRtD  phase 

Project  Definition 

H»JI»tricrlBCAID*ciria> 

t- 

Project  Developnent 
llnplenentation 

. . .  a.alrxir.  cLa.qa 

a-a-cnHO-n-ia-i.  ECP  Jaa^lapnaat. 

Engineering  Validation 

Fleet  Verification 

Fleet  Use 

Logistics 

Readiness  Level 

NIA 

4 

4 

NIA 

NIA 

NIA 

Design 

Interface 

Revieu-andidentifyxiqnificant 

impactx  of  project  to 
exixtinqxyxtem  nr  platform  (ex. 

-a v-ail-ablL-  p-nLJL-r,  LJL-iqht  conrtr-air.hr, 

etc.JCreateFOAMtorexolveanydexiqn 
interface  Itjum. 

Exirtinq  Rtliqlilitr 

revieued.  Initial  improvement 
predictionr  ■di-hL-rrr.ir.L--d.  Duiq> 

For  nL-u  dexiqnx,  Ra-li-ab-ilihx  Ca-ati-rtJ 
Hai.taaaac*  (RCH)  and  Fail. r* 
Hadu  a>d  Effaatr  A>aljxir 
(FHEA)  completed  to  idL-nhif y  failure 
modex,  failure  frequency,  effect  on 
F-L-rfarmanaL-.  and  criticclity.  For 
modificqtionxtDexixtinqdexiqn.RCMand 

Rexultxof  RCH  and  FHEA  uxpd  to 

dL-UL-loF-  or  modify  L-:-:ixtinq  aon-di tion 
baxedandxchedulebaxedmaintenance 

taxkx.  Rexultxof  RCM  and  FMECA  alxo 
uxe-d  to  UF-d at l-  th.L-  Critical  ItL-mx  lixt  ax 
applicable.  Technical  data  updatex 
drafted  and  validated. 

Ta-ckoical  apdatax  (xuch  or 
Maintenance  Requirement  Card 
chanqex)verified. 

Teckaical  updatex 
completed  and  available. 

IE 

Technical  Data 

Oriqinal  Equipment  Manufacturer  hiqher 
level  drauinq,  publication, rpecific-ation 
available 

Applicable  Material  Safety  Data  Sheetr 
(MSDSJ  Available 

P«J<ct  Draui.qlT.ch  Data 
Packaq*  elementr  revieued  an-d 
requiremer.tr  identified  includinq  ax 
applicable: 

Specx/techmanuaLx/enqineerinq 

Safety  Requirementx 

F'rL-XL-r v-ahinr.  an-d  packaqinq 
requirementx 

Textrequirementrdata 

P  r  e  v  l-  n  h  a  h  i  v  e 

Pa-ckaqa  l- 1  l- m l- r. tx  draft  development 
includinq  ax  applicable: 

Specx/techmanuaLr/enqineerinqdrauinqr 

SafetyRequirementx 

Prexervationandpackaqinqrequirementr 
TL-xt  rL-quirL-rT.L-r.hr  data 

Preventative  Maintenance/Maintenance 
RequirementxOardr 

Pradact DrauiaqITach  Data 
Packaqa  elementr  draft 
development  includinq  or  applicable: 
Spear/techmanuaLr/enqineerinq 

SafetyRequirementx 

Prexervationandpackaqinq 

requirement 

TL-xt  rL-quirL-mL-nhr  data 

Preventative 

Maintenance/Maintenance 

Pnd«l  DrauiaqITack  Data 

Packaqa  L-lL-mL-r.hx  updatL-d  includinq 
ax  applicable: 

Specx/tech  m-anuaLx/enqineerinq 

Safety  Requirementx 

Prexervationandpackaqinq 

requirementx 

Text  requirementx  data 

Preventative 

Maintenance/Maintenance 

Pradact  DrauiaqITack 
Data  Packaqe  elementr 
completed  includinq  ax 
applicable: 

Specx/tech 

manuaLx/enqineerinqdxauinqx 
Safety  Requirementx 
Prexervation  and  packaqinq 
requirementx 

Text  requirementx  data 
Preventative 

13 

TacLaical  Diractiaa  M ethodoloq y 
Identif  ied 

Ti-ckaical  Dira-ctiva  draft  in  proeexx 

Tackaical  Dira-ctiaa  draft 
completeandvalidated 

Ttckaical  Dire-ctia*  draf t  verified 
and  updated 

releaxed  and  available 

14 

Tackaical  Pak-licatiaa  jnur-iL  data, 
Rapid  Action  Chanqex  (RADx),  and  Manual 
OhanqL-  Requextx  (MCRxJ  draf tL-d  per 
format 

Tt-ckaical  Paklicatiao  xour-CL 
data,  Rapid  Action  Chanqex  (RACxj  and 
ManualChanqeRequextx  (MCRr) 
validated  for  -zontL-nt  and  format 

Te-ckai-cal  Paklicatiaa 

Inrtructionx  verified  for  content 

Teckaical  Paklicati.a 
(>r  Pak  Ckaaqer  via 

RACx  .r  HCRx)  finalized, 
approved  and  available  for  uxe 

15 

Maintenance 

Planning 

It 

ExirtinqHai>t»aa«  Pla> 

identified  and  revieued 

Haiat<-aaact  Plao  update  drafted  ax 
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LRL  Benefits 


►  Benefits  of  LRLs  include: 

-  Template/ benchmark  to  measure  readiness  by  logistic  element  on  a  project  level  basis 

•  Utilized  to  train/mentor  new  logisticians,  engineering  and  program  management 
personnel  in  sustainment  requirements 

•  Identifies  logistic  risk  areas 

-  Aid  in  planning  manpower/funding/schedule  requirements  for  projects  as  they  mature  from 
project  concept  to  implementation 

-  Dovetail  with  formalized  logistics  risk  assessments  for  another  perspective 

-  Numeric  evaluation  offers  a  way  to  compare  projects  from  a  logistics  readiness  perspective 


Value  of  LRL  is  the  establishing  the 
time  phase  benchmarks 


Booz  Allen  Hamilton 
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Questions? 


Booz  Allen  Hamilton 
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USW  Mission  Capabilities  and  Enabling 
Capability  Integrated  Architecture 


SEA  SHIELD 


Operational  Views  (OVs) 

0.  and 

System  Views  (SVs) 

SV 


FORCEnet 


Mission  Capability  /  Sub-Capabilities 

•  Undersea  Warfare 

-  Provide  Self-Defense 
Against  Subsurface 
Threats 

-  Neutralize 

Submarine  Threats  in  the  Littoral 
Neutralize  Opei^Ocean  Submarine 
Threats 

-  Counter  Minefields  from  Deep  to 
Shallow  Water 

Breach  Minefields,  Obstacles,  and 
Barriers  from  Very  Shallow  Water 
Beach  Exit  Zone 
Conduct  Mining  Operations 

Ofc7 


Enabling  Capabilities  /  Sub-Capabilities 

•  Networks 

-  Comms 

-  Data  T ransfer 


ISR 


-  Information  Processing 

-  Detect,  ID  and  Cue  Target  Information 

-  Assessment 


•COPT 


Operational  Nodes, 
Triggers/Events, 
Operational 
Activities,  System 
Functions,  Etc. 


-  Mission  Planning 

-  Integrate  and  Distribute  Sensor  Date 

-  Environmental  Data 

-  Track  and  Facilitate  Engac 


Capability  Based  Acquisition  Process 


JCIDS  ANALYSIS 
[FAA,  FNA,  FSA] 

JOpsC 

* 
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Architectures 
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Task  Analyses  and  Capability  Assessments 

Analysis  of  Materiel  Approaches 

Refine 
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REFERENCES: 
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3170.01  E,  CJCSI  6212.01C,  DODAF 

NOTES: 

1  -  Initial  IT  Standards  Profile 

2  -  Acronym  List 

3  -  Final  IT  Standards  Profile 


AV-1  OV-2  OV-5  TV-1 
AV-2  OV-3  SV-1 


Architecture 

The  structure  of  components,  their  relationships,  and 
the  principles  and  guidelines  governing  their  design 
and  evolution  over  time 
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ASSET  Products  And  Intent 


Vetted  Integrated  ASW  Mission  Capability  Architecture  (MCA)  for  SoS 
acquisition  alignment  assessments 


Assessments  highlighting  the  impact  on  ASW  mission  capabilities  arising 
from: 

Changes  in  the  Program  of  Record 
Other  Scenarios/Excursions 
Changes  in  SoS  capabilities 


Continuous  Knowledge  Management  and  collaboration 
SoS-based  assessment  tools  and  techniques 
Configuration  managed  ASSET  repository 


To  ensure  that  the  ASW  architecture  and  assessments  are  aligned  with 
program  requirements,  programs  of  record, 
and  on-going  capability  assessments 


Purpose  of  Mission  Capability  Architecture 


The  goal  of  this  MCA  is  to  create  a  common 
denominator  across  the  Joint  and  Coalition 
Enterprise  enabling: 

Understanding, 

Assessment, 

Comparison, 

And  Integration 


of  FoS  and  SoS  interoperability  and  capability 
evolution 


Role  of  MCA  Products  to  ASW  Stakeholders 


Architecture  communicates  desired  capability  to  all 
involved  in  delivering  the  capability 

End  User  (Fleet) 

•  Investor  (OPNAV) 

Material  Providers  (Acquisition  Community) 

Architecture  provides  a  structured  means  for  user 
needs  to  be  translated  into  requirements  describing 
the  material  solutions. 


Success  depends  on  Fleet  involvement 


3rd  Fleet  Operational  Agent  endorse  ASW  OVs 

•  NMAWC  role  as  ASW  ‘voice  of  the  Fleet’ 

ASW  OV  context 

N81  Scenario  alignment  (Major  Combat  Operations) 

•  Global  ASW  CONOPs 

•  DRAFT  OPNAV  ASW  Initial  Capabilities  Document  (ICD) 

Engage  all  ASW  communities  of  interest 


Desired  Capability, 

Right  Information,  Right  Time,  Right  Place 
Resulting  in  desired  effect 
(Operational  Views) 


Operational  View  Products 


Operational  Concept  -  high  level  abstraction  of  the  problem  to  be  solved 
endorsed  by  the  end  user  (Fleet).  Architecture  frameworks  defining  the 
concept  also  serve  as  the  foundation  for  ‘systems’  development  by 
communicating  end  user  desired  capability  to  the  acquisition  community 
charged  with  developing  material  solutions.  Architecture  frameworks  include: 

High  Level  Operational  Concept  Graphic  (OV-1)  -Provides  a  quick,  high-  level  description  of 
what  the  architecture  is  supposed  to  do,  and  how  it  is  supposed  to  do  it 

Operational  Node  Connectivity  Description  (OV-2)  -Tracks  the  need  to  exchange  information 
from  specific  operational  nodes  (that  play  a  key  role  in  the  architecture)  to  others,  however,  it 
does  not  depict  the  connectivity  between  the  nodes. 

Command  Relationship  Chart  (OV-4)  -Clarifies  the  various  relationships  that  can  exist 
between  organizations  and  sub-organizations  within  the  architecture  and  between  internal 
and  external  organizations. 

Activity  Model  (OV-5)  -  Describes  the  operational  activities  and  their  sequencing  that  are 
normally  conducted  in  the  course  of  achieving  ASW  missions 


Systems  View  Products 


Identifying  ASW  systems  support  to  ASW  operations  is  a  three-step  procedure: 
Identify  and  describe  the  ASW  systems,  link  ASW  system  functionality  to  the 
ASW  systems  capable  of  providing  the  functionality,  and  link  ASW  systems  to 
the  ASW  operational  activities  they  support 

SV-4:  Systems  Functionality  Description  -  Identifies  and  describes  the  functional  support 
provided  by  systems  to  ASW  operations 

SV-5:  Operational  Activity  To  Systems  Function  Traceability  Matrix  Depicts  the  mapping  of 
operational  activities  (OV-5)  to  system  functions  and  thus  identifies  the  transformation  of  an 
operational  need  into  a  purposeful  action  performed  by  a  system. 

SV-7:  Systems  Performance  Parameters  Matrix  Specifies  quantitative,  performance 
characteristics  of  systems,  their  interfaces,  and  their  functions. 

SV-8:  Systems  Evolution  Description  Describes  the  evolution  plans  that  describe  how  the 
systems  will  evolve  over  a  lengthy  period  of  time. 


System  Views  to  be  endorsed  by  Virtual  SYSCOM  and  ASN  RDA  CHENG 


Translating  Operational  Requirements 
to  systems  requirements 


Architectures  communicate  desired  capability  to  all 
involved  in  delivering  the  capability  - 

Desired  capability 

Right  Information,  right  time,  right  place 
Resulting  in  desired  effect 
(Operational  Views) 

Architectures  provide  a  structured  means  for  user  needs 
to  be  translated  into  requirements  describing  the  material 
solutions- 

Map  operational  activities  to  systems  functions 

Map  systems  to  functions 

Map  systems  to  protocols 

Map  systems  to  systems  interoperability 

(System  Views) 


System  of  Systems  Definition 


A  set  or  arrangement  of  independent  systems 

that  are  related  or  connected  to  provide  a  given 
capability.  The  loss  of  any  part  of  a  system  will 
significantly  degrade  the  performance  or 
capabilities  of  the  whole  [i] 

If  the  system  of  systems  has  redundancy  and  fault  tolerance, 
then  the  definition  could  be  amended  so  that  loss  of  any 
component  could  significantly  degrade  performance  instead  of 
will  degrade  performance. 


[1]  CJCSI  31 70.01  E,  May  11,  2005 


ASW  SoS  Assessment  to  Support  Acquisition 


Definition  of  ASW  Operational  Situations 
Description  of  ASW  Mission  Capability 

Definition 


Management 


ASW  Mission  Capability 
Packages 

ASW  Operational  Architecture 
ASW  System  Architecture 
Knowledge  Management  Tool 
Perform  Acquisition  SoS 
Assessments 


Knowledge  Management 

Choose  and  install  selected 
product 

Create  ASW  Architecture 
Database 

Institute  Configuration  Control 
Institute  Access  Control 
Develop  User  Manual 


ASW  Acquisition  SoS 
Assessment 

Define  assessment 
requirements 

Develop  operational  concept 
Develop  implementation  plan 
for  operational  concept 


Current  baseline  ASW  Operational  View  Products  (OV) 
Current  baseline  ASW  System  View  Products  (SV) 

Baselining 


Structure  and  Format 
MCPs  • 

Mission  capabilities  view 
produc|  /description 
Mission  Capabilities 
Traceability  to  Operational 
Activities  Map 
U  pdate  Arch  itectu  re 
Database 

Publish  ASW  MCPs 
Current  As-ls  Baseline 
(Leverage  Navy  Existing 
ASW  Mission  Caps) 

Target  To  Be  Baseline 
(Leverage  Navy  Net  Centric 
Environment  and  Mission 
Caps)  • 

Update  ASW  Architecture 
Database 

Publish  ASW  System 
Architecture 


Target  baseline  ASW  Operational  View  Products 
Target  baseline  ASW  System  View  Products 


ReBaselining 


Current  (As-ls)  baseline 
Target  (To  Be)  Baseline 
Update  Architecture 
Database 

Publish  ASW  Operational 
Architecture  • 

Additional  Mission  Capability 
Products 

MCP  by  System  Matrix 
MCP  evolution  roadmap 
Vet  MCP  Systems  product 
with  ASW  Mission  MCP  and 
Systems  Subject  Matter 
Experts  input  • 

ASW  Mission  Acquisition 
Subject  Matter  Experts  input 
Update  ASW  Architecture 
Database 
Publish  ASW  SoS 


Define  of  ASW  MCP  FoS  and  SoS 
Align  with  POR 

Describe  ASW  MCP  capability  evolution 
Documentation 


Definition  of  ASW  Operational  Situations 
Description  of  ASW  Mission  Capability 
Current  baselirfe  ASW  Operational  View  Products 
Target  baseline  ASW  Operational  View  Products 
Current  baseline  ASW  System  View  Products 
Target  baseline  ASW  System  View  Products 
Definition  of  ASW  MCP  FoS  and  SoS 
ASW  MCP  FoS  and  SoS  Alignment  with  POR 
Description  of  ASW  MCP  Capability  Evolution 
ASW  Architecture  Knowledge  Mngmt  System 
ASW  Architecture  Database 
ASW  Acquisition  SoS  Assessment  Plan 


Jul  Aug  Sep  Oct  Nov  Dec  Jan  Feb  Mar  Apr  May  Jun 


Architecture  Knowledge 
Management  System 

Architecture 

Database 


ASW  SoS  Acquisition  Alignment 
Assessment  Strategy  (1/2) 

Creating  an  “As-ls”  ASW  Mission  Capabiiity  Baseline 


Vetted  OVs 
(  2,  4,  5  ) 

Vetted  SVs 
(1,3,  4,  5,  7,8) 

l 


ASW  Mission/ 

ASW 
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ASW  Capabilities 

ArcniiGCiure 

Template 

Description  of 
Strike  Group 
“As-ls”  ASW  Assets 
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Architecture 
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Performance  Metrics 
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► 


Strike  Group 
“As-ls” 
ASW  Mission 
Capability 
Profile 


ASW  SoS  Acquisition  Alignment 
Assessment  Strategy  (2/2) 


SoS  SE  Initial  Tasks  and  Products 

Fleet  inputs  including  endorsement  process  provide  basis  for  engineering  products 


All  View  (AV-1 )  Overview  and  Summary  Information  -  In  the  initial  phases  of  architecture  development,  it 
serves  as  a  planning  guide.  Upon  completion  of  the  architecture,  it  provides  summary  textual  information 
concerning  the  architecture 

(OV-1)  -  High-Level  Operational  Concept  Graphic  -  Provides  a  quick,  high-  level  description  of  what  the 
architecture  is  supposed  to  do,  and  how  it  is  supposed  to  do  it. 

Operational  Node  Connectivity  Description  (OV-2)  -  Tracks  the  need  to  exchange  information  from  specific 
operational  nodes  (that  play  a  key  role  in  the  architecture)  to  others,  however,  it  does  not  depict  the 
connectivity  between  the  nodes 

Organizational  Relationships  Chart  (OV-4)  -Clarifies  the  various  relationships  that  can  exist  between 
organizations  and  sub-organizations  within  the  architecture  and  between  internal  and  external  organizations 

Operational  Activity  Model  (OV-5)  -  Describes  the  operational  activities  and  their  sequencing  that  are 
normally  conducted  in  the  course  of  achieving  ASW  missions. 

SV-5:  Operational  Activity  To  Systems  Function  Traceability  -  Depicts  the  mapping  of  operational  activities 
(OV-5)  to  system  functions  and  thus  identifies  the  transformation  of  an  operational  need  into  a  purposeful 
action  performed  by  a  system. 

SV-3a:  System  by  Platform  Map,  SV-3b:Function  by  System  Map,  SV-3c:  Systems  by  Systems  Association 
Map,  SV-3e:  System  by  Operational  Activity  Map,  SV-3f:  ASW  Systems  of  Systems 


Summary 


ASSET  will  provide  the  following  results: 

An  end-to-end  mission  capability  with  enhanced  integration  and 
interoperability  across  ASW,  using  SoS  artifacts: 

•  Mission  Capability  Architecture 

•  Open  Architecture 

•  Risk  Management 

A  structured  SE  approach,  influencing  acquisition  decisions  to: 
Deliver  ASW  Fleet  products  faster  and  cheaper 
Design  commonality  into  software  and  processing 

SoS  architectures  that  are  adaptive  and  flexible  to  meet  evolving 
Navy  vision  and  varying  operational  circumstances 


Back-up  slides 


Table  1:  Mission  Capability  Architecture  Products 


Name 

Role 

AV-1: 

Overview  and  Summary 
Information 

In  the  initial  phases  of  architecture  development,  it  serves  as 
a  planning  guide.  Upon  completion  of  the  architecture,  it 
provides  summary  textual  information  concerning  the 
architecture. 

AV-2: 

Integrated  Dictionary 

Provides  a  central  repository  for  a  given  architecture’s  data 
and  metadata  and  enables  the  set  of  architecture  products  to 
be  read  and  understood  with  minimal  reference  to  outside 

resources. 

OV-1: 

High-Level  Operational 

Concept  Graphic 

Provides  a  quick,  high-  level  description  of  what  the 
architecture  is  supposed  to  do,  and  how  it  is  supposed  to  do 
it. 

OV-2: 

Operational  Node  Connectivity 
Description 

Tracks  the  need  to  exchange  information  from  specific 
operational  nodes  (that  play  a  key  role  in  the  architecture)  to 
others,  however,  it  does  not  depict  the  connectivity  between 
the  nodes. 

OV-2a: 

ASW  Mission  Description 

Describes  ASW  Missions 

OV-2b: 

ASW  Capability  Description 

Describes  ASW  Capabilities 

Table  1:  Mission  Capability  Architecture  Products,  Cont. 


OV-2c:  ASW  Mission  by  Capability 

Map 

Associates  ASW  capabilities  (OV-2b)  with  ASW  missions 
(OV-2a)  thereby  defining  the  set  of  ASW  capabilities 
required  for  each  ASW  mission. 

OV-2d:  ASW  Capability  by  Operational 

Activity  Map 

Associates  operational  activities  and  their  sequencing  (OV-5) 
with  ASW  Capabilities  (OV-2b) 

OV-2e:  ASW  Mission  Capability 

Profile 

Describes  the  desired  performance  levels  for  the  ASW 
Capabilities  for  each  ASW  mission  supported  (OV-2c) 

OV-4:  Organizational  Relationships 

Chart 

Clarifies  the  various  relationships  that  can  exist  between 
organizations  and  sub-organizations  within  the  architecture 
and  between  internal  and  external  organizations. 

OV-5 :  Operational  Activity  Model 

Describes  the  operational  activities  and  their  sequencing  that 
are  normally  conducted  in  the  course  of  achieving  ASW 
missions. 

SV-la:  Physical  Node  Description 

Describes  the  platforms  and  facilities  and  their  associated 
locations  that  house  ASW  operational  nodes  and/or  ASW 
systems 

SV-lb:  ASW  Systems  Description 

Identifies  and  describes  the  Systems  that  provide  functional 
support  to  ASW  operational  activities 

Table  1:  Mission  Capability  Architecture  Products,  Cont. 


SV-2:  Systems  Communications  Description 

Depicts  pertinent  information  about  communications 
systems,  communications  links,  and  communications 
networks  that  link  the  physical  nodes  identified  in  the  SV-la 
and  SV-lb. 

SV-3a: 

System  by  Platform  Map 

Associates  ASW  systems  (SV-lb)  with  physical  nodes  (SV- 
la) 

SV-3b: 

Function  by  System  Map 

Associates  ASW  functions  (SV-4)  that  support  ASW 
operational  activities  the  ASW  Systems  (SV-lb)  capable  of 
providing  the  functionality. 

SV-3c:  Systems  by  Systems 

Association  Map 

Describes  pair-wise  relations  between  ASW  systems 
including  dependency  of  one  system  on  another,  shared 
interface  protocols,  connectivity 

SV-3e: 

Map 

System  by  Operational  Activity 

Associates  ASW  systems  (SV-lb)  with  ASW  operational 
activities  (OV-5)  they  support  via  their  functionality. 

SV-3f: 

ASW  Systems  of  Systems 

Identifies  the  systems  that  collectively  provide  the  functional 
support  to  an  ASW  capability.  The  System  of  Systems 
definition  is  derived  from  information  contained  in  the  OV- 
2b,  OV-2e,  OV-5,  SV-3b,  SV-4,  and  SV-5  products. 

Table  1:  Mission  Capability  Architecture  Products,  Cont. 


SV-4:  Systems  Functionality  Description 

Identifies  and  describes  the  functional  support  provided  by 
systems  to  ASW  operations 

SV-5:  Operational  Activity  To  Systems 
Function  Traceability  Matrix 

Depicts  the  mapping  of  operational  activities  (OV-5)  to 
system  functions  and  thus  identifies  the  transformation  of 
an  operational  need  into  a  purposeful  action  performed  by 
a  system. 

SV-7:  Systems  Performance  Parameters 
Matrix 

Specifies  quantitative,  performance  characteristics  of 
systems,  their  interfaces,  and  their  functions. 

SV-8:  Systems  Evolution  Description 

Describes  the  evolution  plans  that  describe  how  the 
systems  will  evolve  over  a  lengthy  period  of  time. 
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Toward  Flight  Testing  Less  * 


Presentation  Outline 


Background 
Contest  Winners 

USNI  Anchoring  Sea  Enterprise  Contest 
An  Alternate  Approach 
Analytic  Options 
Summary 


Background 

1943  -  First  helicopter/shipboard  landing 

1945  -  NATC  established  at  Pax  River  MD 

1949  -  Rotary  Wing  Branch  established  in  the  Flight 
Test  Division  of  the  NATC  at  Pax  River,  MD 

1982  -  Dynamic  Interface  Section  established 

1980  -  Deming  -  Total  Quality  Management 

1990  -  Reduce  cost  &  cycle  time  of  T&E 

2000  +  -  Lean,  Six  Sigma,  AIRspeed,  Sea  Enterprise 
(Doing  less,  better,  etc) 


US  Naval  Institute 

Anchoring  Sea  Enterprise  Essay  Contest  (1) 

•  Essay  Contest  announced  in  Jan  2006 

-  3,500  word  essay;  Several  prizes  (lst=  $15K) 

•  Areas  of  Emphasis 

-  Increasing  Efficiency  and  Effectiveness 

-  Increasing  Productivity  (..doing  less,  better) 

-  New  Technology 

-  Accelerating  Innovation 

-  Adaptive  Organization  Design 

-  Outstanding  Communication 

-  Barriers,  Incentives,  and  Mechanisms  Necessary  for  Change 

-  Leading  Change 

•  Received  260  essays 


US  Naval  Institute 

Anchoring  Sea  Enterprise  Essay  Contest  (2) 

•  Winning  essay:  “Sea  Wiki:  How  to  Take  the 
Navy’s  Culture  of  Innovation  to  New  Depths” 

LCDR  Frederick  Dini,  Navy  Supply  Corp 

-  A  Wiki  is  a  web  page  which  contains  all  the  information  on  a  topic 
and  anyone  can  add,  delete,  or  edit  its  content. 

•  2nd  -  “Retire  the  Twenty-Year  System”  Ben 
Atkins,  GE  Energy  Financial  Services 

•  3rd  -  “Sea  Enterprise:  Get  Under  Way  on 
Manpower”  CAPT  Ken  Perry,  Submarine 
Development  Squadron  12 

•  Hon  -  “It’s  the  Network”  ENS  Tim  Graczewski, 
USNR 


One  of  the  260  Essays: 

Technology  Options  to  Enhance  Rotorcraft/Ship 

Testing  and  Related  Analysis 


•  Integrate  design  &  flight  test  analytics 

•  Generic  air  vehicle  model  structure 

•  USNTPS  &  Army  ADS-33PRT  criteria 

•  High  performance  computing  options 

•  Multimedia  test  planning  &  reporting 

•  Intelligent  aircraft  flight  test  database 

•  Advanced  aircraft  simulator  MOE 

•  Aircraft/ship  analytic  options 

•  Advanced  miniaturized  data  systems 


Integrating  Design  &Flight  Test  Analytics 


Flight  Test 
Criteria 


Deterministic 

Stochastic 


USNTPS  Maneuvers 
Army  ADS-33PRT 


FLIGHTLAB  Architecture 
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trail:  »cope-bufts«f lltht lab.coo 

Heading  exponents  file  /hone/’luthcr/coeponcnts.cdf 

Reading  startup  file  /bone/ luttier/startup.exr 
leading  utility  functions... 

Loading  rosponnnt  unthods... 

Loading  solution  techniques. . . 
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PilotStation 

Stef1 


USNTPS  FQ&P  Tests  Modeled  in  the 
FLIGHTLAB  Environment 


ID  Test  Type 

AS 

KGAS 

Test  Conditions 

Hp  OAT 

FT  Deg  C 

Test  Configuration 

Nr  WT  FSCG  BLCG 

RPM  LBS  Inch  Inch 

FCS 

Status 

I/O 

Others  Plot 

Options 

J  Hover 

0 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

j  Critical  Azimuth 

20 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

J  Low  Speed 

0 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

J  Level  Flight 

40 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

j  Climb 

60 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

J  Autorotation 

60 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

J  Coordinated  Turn 

60 

0 

15 

257.8 

16290 

360 

0 

1 

Inputs... 

Results... 

Lng  Stat  Stability 

60 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

J  Lat/Dir  Stat  Stab 

60 

0 

15 

257.3 

16290 

360 

0 

1  i 

Inputs... 

Results... 

J  Maneuver  (push/^ull) 

60 

0 

15 

257.3 

16290 

360 

0 

1  i 

Inputs... 

Results... 

j  Maneuver  Stab  (turn) 

60 

0 

15 

257.8 

16290 

360 

0 

1  i 

Inputs... 

Results... 

j  Control  Response 

0 

0 

15 

257.3 

16290 

360 

! 

L 

0 

Inputs... 

Results... 

4L _ i _ i _ i 

Run 

Reset 

Stop... 

Limits... 

Recover  Results 

Close 

ID  Test  Type 


Test  Conditions 


FCS  Others  Plot 


Test  Configuration 

AS  Hp  OAT  Nr  WT  FSCG  BLCG  Status  Options 

KCAS  FT  Deg  C  RPM  LBS  Inch  Inch  1/0 


Run 


Reset 


Stop. 


Limits 


Recover  Results 


Gose 


Help 


Parallel  Processing 
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FLIGHTLAB  Dynamic  Interface  GUI 


Right  Profile 


Test  Conditions/Configurations 


Stationkeeping  ... 


Approach  ... 


Descent ... 


□ft  off  ... 


Departure  ... 


On- Deck  Handling  ... 


Engage  or  Disengage  ... 


Atmosphere  Model  0:  Standard  Day 

Rotorcraft  Position  0:  Landing  Spot 

Sea  State  Condition  0:  Standard 

Ambient  Pressure  Altitude  [ft] 

58.1393 

Outside  Ambient  Temperature  [degC] 

15 

Rotor  Rotational  Speed  [rpm] 

257.831 

Rotorcraft  C.G.  (Buttline  Station)  pnch] 

0.2004 

Rotorcraft  C.G.  (Fuselage  Station)  pnch] 

354.096 

Rotorcraft  C.G.  (x,y,z)  in  l-frame  [ft] 

Rotorcraft  FCS  on/off  Status 

1 

Rotorcraft  Gross  Weight  pbm] 

19462 

Rotorcraft  Wheel  Height  Over  Deck  [ft] 

26.83 

Sea  State  Number  (Standard) 

4 

Sea  Wave  Period,  -1  =min,0=mean,1  =max 

0 

Sig.  Wave  Height,  -1=min,0=mean,1=max 

0 

Sea:  User- Defined  Sig.  Wave  Height  [ft] 

Sea:  User-  Defined  Wave  Period  [sec] 

Ship  C.G.  (x,y,z)  in  1- Frame  (1C)  [ft] 

0  0  0 

Ship  Course  from  North  (1C)  [deg] 

0 

Ship  Forward  Speed  puiots] 

30 

Ship  Landing  Spot  ID 

1 

Ship  Pitch  Attitude  (1C)  [deg] 

0 

Ship  Roll  Attitude  (1C)  [deg] 

0 

Ship  Turbulence  Intensity  Factor 

1 

Wind  Azimuth  from  North  [deg] 

0 

Wind  Magnitude  (horizontal)  puiots] 

0 

Apply  Gose 


Help... 


PC-Based  VLA  Test  Tool  -  CDI 


PC-Based  VLA  Test  Tool  -  SHAI 


Rotorcraft  Flight  T est  Analytic  Options 
Aircraft/Ship  Models 

•  Helicopters  &  Rotorcraft  (type  Dl  flight) 

•  Rotors,  Fuselage,  Engines,  Flight  control 
systems,  landing  gear  arrangement,  etc 

Aircraft/Ship  Environment 

•  Ship  Airwake/Turbulence 

•  Ship  Motion 

•  Visual  Cues 

Component  Integration 

•  Rotor  wake  and  ship  airwake 

•  Ship  airwake  and  ship  motion 

•  Visual  cues 


Dynamic  Interface  Database  History 
Early  (1960S-1 970s) 

Limited  computer  facilities 

Prevented  widespread  data  storage/search  capability 
1987,  “Toward  Automating  the  Helo/Ship  Dl  Database” 

Called  for  automated,  electronic  databases 

a)  Program  Management  Information  (correspondence, 
personnel  info,  project  status,  etc...) 

b)  Quantitative  Test  Data 

1991  ,  “The  Dl  Database” 

Implemented  1987  “Quantitative  Test  Data"  Database 
Centralized  data  storage  +  flexible  search  criteria 
Facilitated  comparison  with  previous  data 

1997,  “Summary  of  Dl  Database  Files” 

Summarized  all  existing  Dl  database  files,  formats 
NONE  were  linked  together 

2004,  “Intelligent  Aircraft/Ship  Data  Analysis  Options” 

Attempt  to  link  components,  facilitate  searching 
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Helicopter/Ship  Qualification  Testing 

(Les  essais  de  qualification  helicoptere/navire) 


Nielsen  Engineering 
Intelligent  Aircraft/Ship  Database 

•  Dynamic  Interface  expert  Integration 
Environment 

DIXIE 

•  Components 

-  Data  Databases 

-  Knowledge  Databases 

-  Rule-Based  Expert  System 

-  Administration  Server 

-  Remote  Object  Server 

-  HTTP  Server 

-  FTP  Server 


DIXIE  Client 

Knowledge  Database  Results  Window 


American  GNC 

An  Adaptive  Data  Fusion  and  Analysis  System 


Components 


•  Neural  Network 

•  Fuzzy  Logic 

•  Theory  of  Evidence 

•  Bayesian  Network 

•  Clustering 

•  Classification 

•  Data  Consistency  Checking 


Kinematics  based 
Data  Consistency 


The  kinematics  based  consistency  check  of  the  data  relies  on  the 
kinematics  of  the  platform  which  describe  the  characteristics  of 
motion. 

Establishing  the  kinematics  based  consistency  of  the  data  includes 
three  key  steps: 

(1)  Formulating  the  kinematics  model 

(2)  Defining  the  measurements  error  model 

(3)  Using  a  Kalman  filter/smoother  to  verify  the  test 
results  and  isolate  the  invalid  data 


Flight  Testing  Less,  Better 
A  Few  Options 

Develop  the  ability  run  the  flight  test  matrix 
analytically  while  developing  the  test  plan  to 
check  how  close  planned  flights  approach  limits 
and  to  perform  parameter  sensitivity  studies 


Flight  test  planning  and  reporting  require  a  large 
portion  of  time  allotted  to  a  test  program 

-  Advanced  multi-media  test  planning  and  reporting 
options  could  be  used  to  generate  flight  test  matrices 
and  help  reduce  the  test  cycle  time 

-  Design  of  Experiment  options  could  be  used  to  help 
optimize  (minimize)  the  flights  listed  in  the  flight  test 
matrix 

-  A  Wiki  engine  could  be  used  to  consolidate  all  the 
information  associated  with  the  project 


Summary 

The  Concept  of  Flight  Testing  Less,  Better 
can  be  achieved 

The  Concept  of  Flight  Testing  Better, 
Faster,  Cheaper,  and  Safer  can  also  be 
achieved 

It  will  require  improved  flight  test  support 
analytics 

It  will  require  improved  flight  test  support 
databases 

It  will  also  require  integrating  the  design 
and  flight  test  analytics  and  the  related 
databases 

Considerable  progress  has  been  made, 
but  more  is  needed 


Cost  As  An  Independent  Variable 

Balancing  Performance  and  Affordability 


Ed  Casey 
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Areas  to  be  Covered 


1.  Background 

2.  Cost  As  an  Independent  Variable 

3.  Caiv  Metrics 

4.  Lessons  Learned 


To  quote  Sun  Tzu,  The  Art  of  War,  “the  wise  general  in  his 
deliberations  must  consider  both  favourable  and  unfavourable 
factors.  By  taking  into  account  the  favourable  factors,  he 
makes  his  plan  feasible;  by  taking  into  account  the 
unfavourable,  he  may  resolve  the  difficulties.” 


CAIV  and  DTC 


•  Cost  as  an  Independent  Variable  (CAIV)  is  a  management  discipline. 

The  DOD  CAIV  policy  sets  cost  as  a  military  requirement  and 
requires  that  programs  establish  aggressive,  realistic  cost 
objectives  and  to  manage  to  obtain  those  objectives.  Cost 
objectives  must  balance  mission  needs  with  projected  out-year 
resources.  CAIV  balances  cost,  performance,  risk,  and  schedule. 

that  meets 

customer  cost  requirements  through  an  iterative  process  that 
balances  cost,  performance,  and  supportability  while  eliminating 
non-value  added  activity.  DTC  is  an  inherent  element  of  Integrated 
Product  Development  (IPD)  and  is  implemented  within  the  common 
Integrated  Product  Development  System  (IPDS).  DTC  is  the  process 
of  designing  at  product  to  meet  a  cost  requirement. 


DoD’s  Challenge 


DoD’s  CHALLENGE 


HOW  TO  SUSTAIN  AND 
MODERNIZE  FORCES  ON 
DRASTICALLY  REDUCED 
BUDGETS  WITHOUT 
“MORTGAGING  THE 
FUTURE  TO  SURVIVE?” 


The  Death  Spiral 
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Decreased  Readiness  Increased  Maintenance 


CAIV  and  DoD 


•  CAIV  is  a  DOD  acquisition  initiative  applicable  to  all  programs 

•  CAIV  assures  an  affordable  program 

•  CAIV  reduces  program  Total  Ownership  Cost  (TOC) 

•  CAIV  encompasses  Design  To  Cost  (DTC) 

•  Cost  as  an  Independent  Variable  (CAIV):  An  acquisition  management  discipline 
wherein  cost  goals  (constraints)  are  achieved  through  tradeoffs  between 
requirements  and  performance. 

•  Tenets 


•  Additional  development  funding  may  be  considered  an  investment  to  reduce 
production  and  O&S  cost 

•  Government  may  establish  Cost/Performance  IPTs  (CPIPT) 

•  User  participates  in  trade  studies  to  determine  affordable  requirements 

•  Risk  is  included  in  all  estimates 


DTC  &  CAIV  at  Raytheon  Missile  Systems 


•  DTC  and  CAIV  are  blended  into  Business  Development  under  the 
heading  of  Affordability. 

•  Within  the  process  at  RMS: 

•  Defined  cost  targets  are  assigned  to  each  IPT 

•  Focus  is  on  identified  cost  drivers 

•  Cost  vs  performance  tradeoffs  are  conducted  that  lead  to  best 
value  solutions 

•  Metrics  are  determined  and  reported  accordingly 

•  Each  design  choice  is  evaluated  simultaneously  for  both  cost 
and  benefit 

•  CAIV  begins  before  Concept  Exploration  and  remains,  with  DTC, 
vigorous  throughout  product  development 

CAIV  -  Cost  As  an  Independent  Variable 


CAIV  Implementation  and  Flow 


•Cost-Performance  Trade  Studies 
•Action  Plans 
•Incentives 

•Commitment  and  Execution 


Identify 

the 

Toolbox 


•Multiple  Tools 
Available 


•Determine 
Program  Cost 
Goals 

•Allocate  Cost 
goals  to  IPT’s 
•Allocate  to 
lower  levels 
•Evaluate  and 
re-allocate  as 
required 


•Cost 
•Performance 
•Schedule 
•Risk 

•Design  to  Cost 


CAIV  as  a  Management  Control  System 


Management  Control 
Systems  are  put  in 
place  to  direct 
targeted  activity 
toward  achievement 
of  the  desired 
results. 


Management  Control  Process 

CAIV  Process 

Set  goals  and  performance  measures 

Sets  Cost  Goal  as  part  of  Affordability  Plan 

Measures  achievement 

Prepares  current  cost  estimate 

Compares  achievement  with  goals 

Current  estimate  vs.  Affordability  Goals 

Computes  the  variances  as  the  result  of  the  preceding  comparison 

Estimates  system  and  subsystem  variances 

Reports  variances 

Reports  $  Data 

Determines  the  cause(s)  of  the  variances 

Cost  Drivers,  spec,  risk,  etc. 

Takes  action  to  eliminate  variances 

Action  plan:  changes 

Follow-up  to  ensure  that  goals  are  met 

Repeats  at  interval  per  plan 

Iterative  Cost/Performance  Tradeoffs 


Seven  Steps  to  an  Affordable  Design 


—  REQUIREMENTS 


/ 


UNDERSTAND 
CUSTOMER 
COST  AND 
PERFORMANCE 
REQUIREMETS 


Allocate 

Cost 

and  Other 
Requirements 


1.  Understand 


2.  Allocate 


Create 

Balanced  Design 
Approaches 
for  Sub-Products 

3.  Create 

B  Estimate  Cost 

Yield,  Risk, 
i  Performance 


4.  Estimate 


5.  Aggregate 


o 

o 

o 
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(A 
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The  engineer  must  use  the  following 
7  steps  to  execute  DTC: 

1.  Understand  requirements 

2.  Analyze  functions 

3.  Identify  physical  alternatives  / 
allocate  requirements  /  plan  task 
Design  synthesis 
Cost  Modeling  -  Estimation  & 
Rollup 

Evaluate  -  Meet  or  changes 
requirements? 

Select/Formalize  Design 


y  Meets  \ 
■(Requirements 


ACCEPT  DESIGN 
APPROACH  AND 
FLOW 

REQUIREMENTS 
DOWN  TO  SUB- 
PRODUCTS 


6.  Evaluate 


7A  Accept 
OR 

7B  Improve/ 
Iterate 


Plus,  an  often  overlooked  8th  step  to: 
8.  Document  and  report  progress 
towards  meeting  the  cost  goal. 


CAIV  Metrics 

The  Balancing  Act  in 
Practice 


Metrics  and  System  Engineering 


An  often  asked  question  deals 
within  what  role  do  metrics  have 
within  the  System  Engineering 
Community. 


INCOSE 


International  Council  on  Systems  Engineering 


Systems  Engineering 
Measurement  Primer 


A  Basic  Introduction  to  Systems  Engineering  Measurement 
Concepts  and  Use 


Version  1.0 


March  1998 


This  document  was  prepared  by  the  Measurement  Working  Group  (MWG)  of  the  International 
Council  on  Systems  Engineering  (INCOSE).  It  was  approved  as  an  INCOSE  Technical  Paper  by 
the  INCOSE  Technical  Board. 


Metrics 


II 


The  purpose  of  any  metric  is  to  drive  proper  behavior. 

•  Proper  behavior  is  achieved  by  setting,  striving  for,  and  ultimately  reaching  goals. 
A  DTC  metric  is  therefore  one  that  keeps  cost  and  cost  reduction  in  the  forefront. 

•  The  proper  metric  for  DTC  is  one  that  establishes  a  system  cost  goal  for  the 
design  and  that  requires  attainment  of  estimated  production  costs  at  specified 
points  along  a  program  timeline  starting  pre-SDD  and  going  through  production. 

•  By  establishing  cost  goals  for  a  program  (and  its  subsystems)  that  are  time 
phased,  and  constantly  decreasing,  a  program  is  able  to  measure  its  cost 
reduction  effort  toward  the  ultimate  program  cost  goal. 

•  The  DTC  metric  is  measured  as  cost  variance  to  the  required  time-phased  goals. 
Any  variance  to  a  cost  goal  should  precipitate  IPT  action  to  eliminate  the 
discrepancy. 

•  Variances  are  measured  and  reported  at  design  team  meetings  and  program 
reviews.  Efforts  to  eliminate  cost  variances  (the  proper  behavior)  become  part  of 
the  IPT  design  effort  when  tradeoffs  are  made  between  cost,  risk,  performance, 
and  cycle  time. 


CAIV  Metrics 


CAIV  Metrics  encompass  not  only  cost,  but  performance,  schedule  and  risk 
as  well.  The  primary  metric  to  measure  specific  CAIV  project 
effectiveness  is  cost.  The  utilization  of  this  metric  requires  an  established 
cost  baseline  in  sufficient  detail  to  compare  prior  and  resultant  impacts  of 

a  CAIV  project. 


Critical  to  the  utilization  of  these  primary  metric  comparisons  is  the  need  for 
an  established  baseline.  Without  it,  comparison  is  meaningless. 


CAIV  Metric 

Threshold 

Goal 

Current 

%  Current/Goal 

Plan  of  Action 

Cost 

Performance 

Schedule 

Other  metrics  that  are  used  to  evaluate  how  well  a  program  is  implementing 
CAIV: 

Establishment  of  the  CAIV  Team  (PM,  User  and  Contractor) 

Office  personnel  familiarity  with  the  CAIV  technique 
Availability  of  CAIV  guidance  Number  of  CAIV  Trade  Studies 

CAIV  accomplishments  CAIV  database 


CAIV  Metrics  —  An  Example 


CAIV  Metric 

Threshold 

Goal 

Current 

Current/Goal 

Risk  Assess 

Cost  Driver 

Latency 

Plan  of  Action 

Cost  -  System 

$ 

32,775.00 

$  31,500.00 

$  37,790.00 

1.20 

Sub-System 

$ 

5,000.00 

$  4,500.00 

$  6,200.00 

1.38 

2 

no 

Sub-System 

$ 

1,500.00 

$  1,500.00 

$  1,400.00 

0.93 

4 

no 

Sub-System 

$ 

12,275.00 

$  12,000.00 

$  17,890.00 

1.49 

1 

yes 

Sub-System 

$ 

8,000.00 

$  7,500.00 

$  6,000.00 

0.80 

6 

yes 

Sub-System 

$ 

2,500.00 

$  2,500.00 

$  2,700.00 

1.08 

3 

yes 

Sub-System 

$ 

3,500.00 

$  3,500.00 

$  3,600.00 

1.03 

2 

yes 

Sub-System 

Sub-System 

(Performance 

Requirement 

Goal 

Current 

Req/Current 

Risk  Assess 

Cost  Driver 

Latency 

Plan  of  Action 

speed  mph 

200 

220 

180 

1.11 

1 

no 

range  nm 

500 

550 

575 

0.87 

1 

no 

load  lbs 

750 

750 

900 

0.83 

1 

no 

KPP-4 

(Schedule 

Contract 

Goal 

Expected 

Exp/Con 

Risk  Assess 

Cost  Driver 

Latency 

Plan  of  Action 

□ 

Dates 

□ 

Months  to  Go 

18 

15 

15 

0.83 

2 

no 

How  much  information  can  you  get  from  this  chart? 

Where  would  you  concentrate  efforts?  What  might  you  want  to  re-visit? 


CAIV  Metrics  Sample  Chart 


CAIV  Metrics  chart  at  a  glance: 

•  Discloses  a  program’s  status  in  the  areas  of  cost,  performance 
and  schedule.  From  the  above  sample  chart  one  can  quickly  see: 

•  The  program  is  projected  to  over-run  costs  by  20%. 

•  Two  of  the  sub-systems  are  in  the  red;  one  with  a  high  risk  of 
failing. 

•  The  PM  has  no  plan  of  action  to  fix  one  of  the  red  areas 

•  One  sub-system  is  in  the  “violet”  with  low  risk  of  failure  so 
perhaps  cost  goals  ought  to  be  re-allocated. 

•  The  others  are  close  to  goals  on  one-side  or  the  other 

•  Two  of  the  performance  areas  have  superseded  requirements 
while  one  area,  without  a  plan  of  action  and  at  high  risk  of 
failure  is  in  the  red. 

•  And,  the  program  is  planning  on  an  early  delivery. 

•  The  color  coding  helps  management  key  in  on  specific  areas  of 
concern  and  make  necessary  changes. 


RECURRING 


REC  Sub-Total 


RECURRING  COSTS 
Management  Report 


Cost  Units:  I 
Date:  [ 


^Assembly  Description 


Recurring 

TARGET 

Cost 


Preliminary  DTUPC  data  prepared  by 


Unit  Production  Quantity: 


Program  Assumptions 


Total  production  quantity: 
Production  rate: 
st  production  contract  quantity: 
Production  factors  similar  to: 

Year  dollars: 


Other  Production  Assumptions: 


Major  Cost  Drivers: 


Unit  Productbn  Quantity: 


Program  Assumptions 


Production  Assumptions: 

Total  production  quantity: 

Production  rate: 
1st  production  contract  quantity: 
Production  factors  similar  to: 

Year  dollars: 


Other  Production  Assumptions: 


I 

Action  Items 

Who 

Assigned 

Completed 

1 

2 

3 

4: 

5: 

6: 

7: 

8: 

9: 

nn 

Action  Items 

Who 

Assigned 

Due 

Status 

CAIV  and  DTC  Reports 


CAIV  Cost  History 

Planning  Curve 

10 

_ ^ 

Program  Name:  1  Program  Name  1 

Assy  or  Rollup: 

|  Design  trade  study/program  events 

Date 

Threshold 

Cost  target 

Cost  plan 

Cost  est. 

Status 

Cost  target:!  $7,500  1  0 

Baseline  Design 

01/01/03 

$8,250 

$7,500 

$9,475 

9,475.0 

1.000 

IcostThreshold^^^SS^^SO^^I  1 

$8,250 

$7,500 

$9,870 

0.000 

|  2 

$8,250 

$7,500 

$9,607 

0.000 

3 

$8,250 

$7,500 

$9,343 

0.000 

4 

$8,250 

$7,500 

$9,080 

0.000 

5 

$8,250 

$7,500 

$8,817 

0.000 

6 

$8,250 

$7,500 

$8,553 

0.000 

7 

$8,250 

$7,500 

$8,290 

0.000 

8 

$8,250 

$7,500 

$8,027 

0.000 

9 

$8,250 

$7,500 

$7,763 

0.000 

10 

$8,250 

$7,500 

$7,500 

0.000 

11 

$8,250 

$7,500 

$7,500 

0.000 

|  12 

$8,250 

$7,500 

$7,500 

0.000 

History  T race 


■Threshold 
-Cost  target 
-Cost  plan 
Cost  est. 


Program,  System,  Component  History  Charts 


II 


Lessons  Learned 


Lessons  Learned 


•  Affordability  is  the  primary  driver  in  all  architecture  design  and 
development  activities. 

•  CAIV  requires  mandatory  cost  requirements  be  assigned  to  all  programs 
down  to  the  lowest  levels. 

•  Programs  must  track  and  measure  their  current  design  to  cost  status 
against  their  goals  at  periodic  intervals.  (Cost  Management) 

•  Cost  must  be  a  design  requirement  with  importance  equal  to  or  greater 
than  performance. 

•  Cost  estimation  can  be  approximate  in  early  program  phases, 
progressively  better  during  later  phases. 

•  Proper  behavior  is  achieved  by  setting,  striving  for,  and  ultimately 
reaching  goals.  CAIV  metrics  therefore  keep  cost  and  cost  reduction  in 
the  forefront  of  IPT  activity. 

•  By  establishing  cost  goals  for  a  program  (and  its  subsystems)  that  are 
time  phased,  and  constantly  decreasing,  a  program  is  able  to  measure  its 
cost  reduction  effort  toward  the  ultimate  program  cost  goal. 


Any  Questions? 


Now  is  a  good  time  to  ask. 


9th  Annual  NDIA  Systems  Engineering  Conference,  Oct.  25,  2006 
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MITRE 

What  is  “Real  Options”? 

■  An  option  confers  a  right,  but  not  obligation,  to  take  an  action  in  the 
future  for  managing  an  asset. 

■  The  Real  Options  methodology  is  a  framework  for  valuing  and 
planning  of  real  assets. 

■  Examples  of  real  options: 

-  A  stronger  foundation  and  structure  for  a  multistory  parking  garage 

-  A  rocket  with  extra  fuel  on  each  satellite  to  reconfigure  a  constellation 

-  Application  “hooks”  built  into  the  architecture  of  a  software  system 

-  A  foundation  IT  asset  enabling  future  high-value  applications 

-  Pilot  projects,  feasibility  studies,  and  prototypes  can  all  create  options 
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Real  Options  Triad 
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Uncertainty 


1 


Investment 
Value 


r 


Flexibility 


The  Return  on 
Investment  can  only  be 
known  probabilistically. 


Strategy 


Viewing  investment  value  through  a  Real  Options  lens: 

The  value  of  a  project  must  be  assessed  not  just  from 
the  technical/engineering  aspect,  but  also  on  how  the 
management  would  dynamically  respond  to 
uncertainties  to  achieve  better  Return  on  Investment. 


MITRE 

Real  Options  supports  strategic  intuition  with 

analytical  rigor 

■  The  traditional  investment  valuation  tends  to  be 

Optimistic  :  assume  the  project  will  finish  and  achieve  optimal  value 
Simplistic:  model  uncertainty  by  an  “average  scenario ” 

Deterministic:  can’t  handle  scenario-dependent  cash  flows  caused  by 
optionality 

■  Through  a  Real  Options  lens,  the  risk  and  strategy  context  of  the 
project  is  examined;  potential  evolution  paths  are  accounted  for. 

The  value  of  an  investment  is  assessed  probabilistically. 

■  4  major  methods  for  Real  Options  Valuation: 

-  Black-Scholes  formula 

-  Binomial  lattice  model 

-  Decision  tree  analysis 

-  Monte  Carlo  simulation 
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Real  Options  can  offer  a  flexible  systems 
engineering  approach  for  capability  acquisition 

■  Consider  these  concepts: 

-  Field  operationally  acceptable  capability  earlier  and  make  evolutionary 
increments  over  time.  Considered  contingency  plans  and  exit 

opportunities.  (Defense  Acquisition  Performance  Assessment  Report,  2006) 

-  Take  evolutionary  steps  to  increase  learning  of  a  product’s  usefulness 
and  consider  an  option  to  terminate  a  project  if  it  is  no  longer  beneficial. 

(GAO-04-744,  2004) 

-  Structure  major  acquisitions  into  useful  segments  with  a  narrow  scope 
and  brief  duration,  (omb  circular  a-ii,  2005) 

■  How  would  you  assess  the  value  of  a  project  being  shaped  by  these 
concepts? 

-  We  use  a  case  study  with  notional  data  to  demonstrate  an  analysis 
methodology  based  on  a  Real  Options  approach. 
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Case:  Improving  Tactical  Data  Link  (TDL)  systems  to 
support  the  Close  Air  Support  (CAS)  mission 


CAS  Mission  Profile 


C2i 

CAS-  €2 


©  Mission 
Assignment 


©  Transit 
to  Target 


^--r  Er\d  of 

'  P  Mission 


CAS  Aj^rin 


0  Termfnaf 
Control 


CilCAS  C2 


0  CAS  Request 


OW.  A  di'vjry 


Supported 


Reqtiifenwnl 

identified 


TAC 


Two  major  problems  in  current  TDL  systems  for  CAS: 

-  Lack  common  data  communication  medium  for  all  participants 

-  Need  more  effective  message  contents  and  delivery  protocols 
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2  TDL  Solutions  for  the  CAS  Mission 


X :  existing  or  programmed  capabilities 


Proposed  solutions: 


X.  Participants 

Primary  CAS 
aircraft 

Secondary  CAS  aircraft 

Joint 

Terminal 

Attack 

Controller 

(JTAC) 

CAS 

Systems  \. 

A- 10 

F-16 

C+ 

F-16 

CG 

F/A-18 

AV-8B 

Gateway 

Mo  de  m+AP  APD 

S 

S 

Modem+VMF 

(A) 

(A) 

S 

S 

(b) 

Modem+MTS 

S 

Situational 
Awareness  Data 
Link  (SADL) 

✓ 

✓ 

Link  16 

S 

(b) 
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Two  Solution  Strategies 
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■  Solution  Strategy  A  :  Equip  primary  CAS  participants 
with  a  common  data  communication  device 

-  A1:  Install  improved  data  modems  with  VMF  message  format  on 
all  primary  CAS  aircraft. 

-  A2:  If  A1  is  not  feasible,  provide  a  light-weight  SADL  device  to 
each  tactical  air  controller. 

■  Solution  Strategy  B  :  Use  CAS  gateways  to  translate  and 
forward  messages  for  all  CAS  participants 

-  Develop  and  field  CAS  gateways;  extend  the  existing  TADIL-J 
message  standard  and  implement  on  CAS  aircraft. 
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Staged  Development  and  Implementation 


Either  Modem/VMF 


Solution 

V  A  y 

N.  y'  \ 

Or  Lite  wt.  SADL 
(A2  path) 


(A1  path) 

_  / 

/ 

Feasibility  study  / 
and  approval  \ 


Phase  1: 

Phase  II: 

Phase  III: 

Coord.  &  prep. 

Basic  modem  capability 

Adv.  modem  capability 

(Ml  phase)  (M2  phase) 


Phase  1: 

Phase  II: 

Pilot  testing 

Implementation  and  integration 

Gateways  +  TADIL-J  extension 


Phase  1: 

Phase  II: 

Develop  CAS  gateways; 

Field  gateways; 

Change  message  standards 

Implement  improved  messages  on  all  platforms 
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Uncertainties  in  TDL  Capability  Acquisition 


Interface  change 
proposal; 
message 
standard  change 


Technology 

Readiness 


Configuration 

management; 

system 

integration 


Extra  funding 
required 


Upgrade 
schedules  of 
platforms 


Schedule 

stretching 


Program 

restructuring 
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Which  solution  should  be  chosen? 

■  Conventional  approach:  trade  off  benefit,  cost,  and  schedule;  use 
sensitivity  analysis  or  scenario  analysis  to  understand  the  impact  of 
uncertainties. 

■  The  conventional  approach  ignores  possible  actions  that  could  be 
taken  by  the  manager  to  dynamically  respond  to  uncertainties. 

■  Our  remedy: 

-  Use  Decision  Tree  to  model  the  interplays  between  technical 
development  and  management  actions. 

-  Use  flonte  Carlo  simulation  to  compute  scenario-dependent 
benefit,  cost,  and  schedule. 
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Starting  off  a  Decision  Tree  for  Solution  A 


Solution  Strategy  A  -  either  Modem/VMF  or  SADL 
A1 :  Modem/VMF 

Ml  -  Provide  standalone  capabilities  for  primary  CAS  aircraft  pilots  to  receive  digital  9-Line  briefing 
M2  -  Digitally  integrate  the  9-Line  briefing  with  the  aircraft  Operational  Flight  Program  (OFP) 

A2:  SADL 

Develop  and  field  light-weight  SADL  to  JTAC  with  suitable  TACP  system  interface  to  enable  direct 
connectivity  to  SADL  aircraft. 


(Analytica  screen  shot) 
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Anatomy  of  a  Decision  Tree 

Each  solution  strategy  is  modeled  as  a  decision  tree  containing  a  series  of 
chance  nodes  and  decision  nodes.  Each  path  of  the  tree  ends  at  a 
terminal  node. 


Chance  Node 

Decision  Node 

Terminal  Nodes 

Activity  Probability  of  outcome  Outcome 


MITRE 


Decision  Tree  for  the  A1  (Modem/VMF)  Branch 

This  decision  tree  is  organized  around  two  kinds  of  uncertainties  considered 
in  tandem.  Each  outcome  is  followed  by  one  or  more  decision  options. 


A1  solution  path  (ModemA/MF) 

Phase  I.  Coordination  and  preparation 
Phase  II.  Acquiring  initial  capability  Ml 
Phase  III.  Acquiring  advanced  capability  M2 


Options 


Options/ 


Options 


60% 


Ready 
for  Ml 


Start 

Phase  I  in 
period  2 


40% 


Not 

Ready 


(Analytica  screen  shot) 


ptions 


Options 


Reduce 

Produt 
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The  Core  Module  of  the  A1  Decision  Tree 

■  Each  phase  may  span  across  multiple  time  periods. 

■  Progress  status  and  outlook  are  reviewed  after  each  period. 

■  The  manager  will  decide  which  option  to  take;  the  project  could  embark 
on  a  different  course  of  action. 


MITRE 


Progress  status  probabilities  can  be  easily 
derived  from  the  probability  of  duration  time 


Duration  time  T of  each  phase  has  a  probability  distribution.  Every  project 
planning  must  have  estimated  this  distribution.  Sample  Probability 

Distribution  of  T 


Start  phase  x  in 
period  1 


P(T  >  2  years) 


(Every  period  =  2  years) 


Continue  phase  x 
in  period  2 


Therefore,  all  these  finished/not- 
finished  probabilities  are  known  data 
to  feed  into  the  decision-tree  model. 

No  guesswork  is  needed  ! 


P(T  >  4  years) 
P(T  >  2  years) 
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Exit  Criteria  for  the  A1  Solution 
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•  The  entire  effort  should  not  take  more  than  6  periods. 

•  Ml  development  should  not  take  more  than  3  periods.  Can 
wrap  up  the  Ml  effort  with  reduced  requirements  (Plan  B). 


•  M2  development  should  not  take  more  than  2  periods. 

Subtree  1:  Ml  production  +  start  M2  in  period  4 
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Decision  Tree  Valuation 

A  decision  tree  is  evaluated  starting  from  terminal  nodes 
back  to  the  root. 

Each  node  keeps  a  value  vector  (b,  c),  which  represents 

the  benefit  and  cost  rolled  back  from  all  terminal  nodes  that 

can  traverse  back  to  this  node.  Terminal  Nodes 


MITRE 


Valuation  at  a  Production  Node 


Let  m  denotes  the  length  of  the  planning  horizon,  and  the  project  takes  n 
years  to  reach  a  Production  node,  then  the  value  vector  at  this  node  is: 


(  m-n- 

V  i= 0 


product  benefit  per  year 
(l  +  benefit  discount  rate)' 


1  O  &  M  cost  per  year 
£4  ( 1  +  cost  discount  rate)'  y 
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Discount  Rates 


■  Cost  discount  rate:  a  measure  of  the  time  value  of  money  for 
investments  and  expenses. 

■  Benefit  discount  rate:  to  express  the  urgent  need  for  a  timely 
solution  or  to  penalize  a  delay  in  delivering  required  capabilities. 

(OMB  Circular  A-4  has  an  example.) 

■  With  the  use  of  discount  rates,  time  preference  is  embedded  in  the 
decision  rule  -  an  alternative  branch  will  be  chosen  if  it  has  a 
higher  ratio  of  discounted  benefit  /  discounted  cost. 

-  Performance,  affordability,  and  timeliness  are  all  molded  into  a  single 
metric  for  solution  comparison. 
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Decision  Tree  Valuation  (cont.) 


Can  also 
include 
constraints  on 
budget, 
schedule,  etc. 


Decision  rule:  choose  the  branch 
having  the  best  benefit/cost  ratio 

Decision  Node 


v 


Activity 


v6  =  Value  of  this 
activity  +  v5 
discounted  back 
one  period 


v6 

Start  Ml  in 
period  3 


40% 


Ml  Not 
Finished 


Chance  Node 


m-n-l 

5  (l  +  r)' 
Terminal  Node 


Ml  Production 


Subtree  1:  Ml 
production  +  start 
M2  in  period  4 


v2 


Simulation: 
x  =  Bernoulli(40%) 
v5  =  x  *  v3  +  (1-x)  *  v4 
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input  data  are  notional 


Comparing  Benefit  and  Cost 
of  Solutions  A  and  B 
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Their  values  can  only 


fl 

Benefit 

A 

B 

Min 

0 

0 

Median 

771 

871 

Mean 

747 

867 

Max 

880 

1122 

Std.  Dev. 

138 

223 

be  known  probabilistically. 


1} 

Cost  ($M) 

A 

B 

Min 

5.7 

9.7 

Median 

21.1 

31.1 

Mean 

21.0 

31.0 

Max 

22.0 

32.6 

Std.  Dev. 

2.0 

2.7 

No  clear  winner. 

B  may  get  better  benefit  but  at  higher  cost. 
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input  data  are  notional 
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Comparing  Solutions  A  and  B 


based  on  Benefit  /  Cost  Ratio 


Return  on 
Investment  (ROI) 


Benefit  /  ^ 
Cost  ratio 

A 

B 

Min 

0 

0 

Median 

37 

28 

Mean 

35 

28 

Max 

40 

34 

Std.  Dev. 

7 

7 

We  are  90%  certain  that 
A’s  ROI  value  would  exceed  32; 
B’s  ROI  value  would  exceed  22. 


A 

B 


Value-at-Risk  (VaR)  Graph 
a.k.a.  Risk  Profile 


Conclusion:  With  all  potential  outcomes  considered, 

A  is  probabilistically  better  than  B  for  Return  on  Investment. 
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Cumulative  Probability 
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Value-at-Risk  Graph  Magnified 


A 

B 
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Another  Look  at  the  Value-at-Risk  Graph 
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Uncertainties  in  project  development  and  funding  decision 
have  been  translated  into  Risk  in  ROI. 


Which  solution  has  a 
higher  risk  of  failing  to 
achieve  a  desirable 
level  of  ROI? 


_  _  I 
I 


P(  B’s  ROI  <  z  )  >  P(  A  s  ROI  <  z  ) 
for  any  z  >  0 


3 


16  25 

Benefit  f  Cost  Ratio 


33 


41 


Key 


Solution  strategies 

A 

B 


Solution  B  will  have  a  higher  probability 
of  failing  to  achieve  any  desirable  level 
of  ROI.  Hence,  Solution  A  is  better. 
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Options  can  enhance  Solution  A’s  ROI 
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.  .  .  Base  option 

.  .  .  Option 
to  stop 


.  .  .  Base  option 


.  .  .  Option 
to  reduce 


ROI 

A  without 
options 

A  with 
options 

Min 

0 

0 

Median 

28 

37 

Mean 

29 

35 

Max 

38 

40 

Std.  Dev. 

7 

7 

o 


■ii 

i- 

is 

□ 

E 

D 

O 


1  I 


0  + 
0 


Options  shift  the 
Risk  Profile  to  the 
right.  Good! 


-i - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 

4  8  12  16  21  25  29  33  37  41 

Benefit  /  Cost  Ratio 
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A  with  option 
A  without  option 
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Possible  Outcomes  of  Solution  A 


options 


Probability 

37% 

19% 
16% 
12% 
7% 
3% 
3% 
1% 
1% 
0% 


#  of  years  spent  to  get  the  product 
2  4  6  8  10 
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Without 

options 

Longer  time 
Lower  ROI 


5% 

3% 

3% 

2% 

1% 

1% 

1% 

0% 

0% 


32.8 

29.5 

33.2 
0 
0 
0 

28.3 


Product  ROI 
Ml 

Ml 

SADL 
Ml 

Reduced  Ml 
Reduced  Ml 
SADL 
None 
None 
None 

M2 
M2 

SADL 
M2 

Reduced  Ml 

Ml  28.2 

SADL  33.2 

Reduced  Ml  24.9 
Ml  30.4 

None  0 

None  0 

None  0 

None  0 

Ml  32.7 
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Strategy  Analysis  for  the  “M2  option” 
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Ml 

Ml  production 

finished 

&  M2  development 

Ml  production  /|| 


Under  what  conditions  would 
the  “M2  option”  become 
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M2  annual  benefit  value 


strategy  map 
derived  from 
sensitivity  analysis 


■  Most  critical  factor:  M2  starting  investment 

■  CZ>  current  estimates  of  M2  starting  investment  and  yearly  benefit 

■  Green  zone:  favorable  conditions  for  taking  the  M2  option 

Based  on  the  given  data,  the  M2  option  is  unlikely  to  be  exercised. 
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Strategy  Analysis  for  the  “full-scale  Ml  option” 


Reduce 
Ml  rqmt  to 
finish  in 
period  4 


Reduced  Ml 
Production 


60% 

Ml  Not 

Finished 

How  likely  will  this 
“full-scale  Ml  option” 
be  exercised? 


Ml  Productiony 
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production  h 


Ml 

start 


(this  M2  option  will 
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Ml  in 
period  4 
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It  can  be  proved  that  the  above  decision  tree 


If  the  upper  branch  is  more 
cost  effective,  then  p%  =  0%; 
otherwise,  p%  =  probability  of 
completing  Ml  in  period  4. 

Based  on  the  given  data,  the  lower  branch  is  more  cost  effective,  so  there 
is  a  75%  chance  that  the  “full-scale  Ml  option”  will  be  exercised  in  period  4.  29 


can  be  transformed  into: 


Further  Sensitivity  Analyses  on 
Discount  Rates  and  Planning  Horizon 


A  is  increasingly  better  than  B  when  the  cost  discount  rate 
increases. 

As  advantage  over  B  gets  diminished  as  the  benefit  discount  rate 
increases 

-  When  benefit  discount  rate  >  19%,  B  becomes  the  preferred  solution. 

-  The  benefit  discount  rate  models  “time  preference”  or  urgency  for  a 
solution.  If  you  want  a  solution  so  “bad”,  B  might  be  a  better  choice. 

A  is  increasingly  better  than  B  for  longer  planning  horizon.  If  it’s 
shorter  than  16  years,  there  is  no  clear  winner. 


MITRE 

A  solution  selected,  a  strategy  suggested 


Gateways  +  TADIL-J  extension 


/Solution'', 

i  ' 

.  • 


gghase  ya 

Develop  CAS  gateways; 
Change  message  standards 


Phase  II: 

Field  gateways; 

Implement  improved  messages  on  all  platforms 


Strategy: 

-  If  the  A1  path  is  feasible  to  go,  just  develop  the  basic  capability  Ml . 

-  May  need  to  consider  reduced  Ml  development  (as  Plan  B). 

-  After  each  period,  reassess  the  strategy  with  most  current  data. 
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Real  Options  Thinking  for  the  TDL-CAS  Case 


■  Uncertainty: 

-  Convoluted  schedule  risk  in  system  and  platform  upgrade 

-  Technology  readiness 

-  Funding 

■  Flexibility: 

-  Deliver  operationally  acceptable  capability  in  near  term 

-  Prepare  to  acquire  capability  incrementally 

-  Consider  contingency  plans  and  exit  opportunities 

■  Strategy: 

-  Field  initial  capability  and  give  up  further  development 

-  Reduce  requirements  and  wrap  up  effort  after  n  years 
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Conclusion 

■  Managerial  flexibility  can  make  significant  difference  in 
investment  value  of  capability  acquisition  programs. 

■  Decision-tree  analysis  and  Monte  Carlo  simulation  are 
useful  tools: 

-  Decision  trees  can  model  flexible  systems  engineering 
concepts.  The  Decision  Maker  will  be  well  informed  of  decision 
consequences.  The  decision  tree  should  be  a  live  one  with 
refreshed  data  every  period  to  provide  updated  advice. 

-  Monte  Carlo  simulation  with  risk  profile  analysis  enables 
probabilistic  evaluation  of  Return  on  Investment. 
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Solution  benefit  is  estimated  from 
a  multi-attribute  value  analysis 
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input  data  are  notional 
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Probabilistic  Evaluation  of  Solution  A 


We  are  90%  certain  that  A  s... 
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Probabilistic  Evaluation  of  Solution  B 


We  are  90%  certain  that  B’s.. 
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Possible  Outcomes  of  Solution  A  with  Options 


ROI 

Project  Years  and  Production  38 


Probability 
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Possible  Outcomes  of  Solution  A  without  Options 


This  analytic  approach  can  be  applied  to 
projects  with  similar  characteristics 


MITRE 


■  There  exist  significant  uncertainties  in  project  development  and 
funding  decisions. 

■  There  is  time-to-market  pressure,  but  the  product  development 
process  will  be  long  and  has  multiple  phases  with  uncertain  duration 
in  each. 

■  An  initial  useful  capability  can  be  defined  and  it  can  enable  the 
development  of  more  advanced  capabilities. 

■  The  project  is  not  destined  to  acquire  a  “100%”  solution;  contingency 
plans  and  exit  strategies  are  allowed  and  encouraged. 
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Overview  of  Presentation 


•  Simulation-Based  Acquisition  (SB A) 

•  Design  Issues 

•  Case  Study 

-  Virtual  Systems  Integration  Lab  for  the  Army 

•  Recommendations 

•  Conclusions 

•  Questions? 
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Simulation-Based  Acquisition 

circa  1998 

•  Definition  from  U.S.  ARMY  PEO  STRI  (1998)*: 

-  "Simulation  Based  Acquisition  (SBA)  is  a  major  initiative. .  .intended  to 
make  smart  use  of  M&S  technologies  to  equip  our  forces  with  quality 
systems  of  high  military  worth,  in  less  time,  and  at  lower  cost  than 
traditional  means. 

“The  concept  behind  SBA  is  that  the  M&S  tools  can  be  integrated  and 
matured  throughout  the  system  lifecycle  process  ...” 

•  Overall  Goal:  Field  the  best  systems  for  the  future  military  force  in  the 
shortest  time  and  at  the  lowest  cost. 


*http://www.peostri.  army.mil/PRODUCTS/SBA/defmition.jsp 
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Simulation-Based  Acquisition 

circa  2006 

•  We  have  not  achieved  full-adoption  of  SB  A  across  the  DoD 

•  SBA  success  has  been  limited 

-  OSD  study  suggests  that  over  80%  of  models  developed  on  22  major 
programs  are  unique  to  that  program. 

-  Most  are  owned  by  the  prime,  not  the  government 

-  Leads  to  increased  development  time 

-  Leads  to  increased  cost  to  the  government 

•  We  remain  on  the  ground  floor  of  SBA 

•  The  Overall  Goal  is  still  the  same 
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What  are  the  Design  Issues? 


•  Two  major  kinds:  Technical  and  Political 


•  Both  are  co-dependent  but  can  be  analyzed  separately 

•  Both  present  thorny  challenges,  but  are  not  insurmountable 
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Technical  Challenges 


Government  Perspective 

Industry  Perspective 

-Need  high  fidelity  models 
-Need  data  security 
-Need  system  of  systems 
-Need  interoperability 

-Need  accurate  data 
-Need  IP  protection 
-Need  subsystem  requirements 
-Need  cooperation 

•  SUMMARY :  We  need  a  solution  for  the  Govt  to  maintain  control  of 
model  assets,  while  providing  Industry  the  data  it  needs  to  do  its  job. 
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Technical  Challenges 


•  Main  challenge  is  data  access 

•  Model  validation  cannot  be  done  without  access  to  accurate  data 

•  The  current  workaround  is  to  maintain  isolated  model  repositories 

•  However,  this  workaround  defeats  interoperability  and  cooperation 

•  There  is  no  unified  tool  for  virtual  systems  integration 

•  The  result  is  that  full  systems  cannot  be  simulated  reliably  or  at  all 
under  current  resource  constraints 
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Political  Challenges 


Government  Perspective 

Industry  Perspective 

-Want  to  own  models 

-Want  more  control 

-Want  best  solution 

-Want  more  cost-savings 

-Want  to  own  models 
-Want  job  security 
-Want  less  competition 

-Want  fewer  restrictions 

•  SUMMARY :  We  need  a  solution  that  allows  the  sharing  of  simulation 
models  in  a  way  that  does  not  penalize  or  overly  burden  the  creators. 
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Political  Challenges 


•  Main  challenge  is  simulation  reuse 

•  The  Govt  will  not  get  cost-effective  SBA  without  model  reuse 

•  The  biggest  Industry  players  stand  to  lose  the  most  from  SBA 

•  The  current  environment  breeds  specialized  simulations  of  limited 
reuse  that  are  costly  to  build 


•  Simulation  reuse  and  interoperability  is  painful  for  industry  to 
implement,  and  will  not  happen  without  greater  Govt  mandate 
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Other  Challenges:  Myths  of  SB  A 


•  Myth  1 :  Job  security  is  endangered  by  SBA. 

-  Truth:  Existing  designers  are  given  new  tools  to  do  their  jobs  better. 
Potential  for  new  service  contracts  are  created  by  model  maintenance. 

•  Myth  2:  SBA  processes  will  “break  the  system.” 

-  Truth:  The  current  system  is  unlikely  to  accomplish  all  DoD 
transformation  goals  under  shrinking  budgets.  The  current  system  has 
mission-critical  problems  and  SBA  is  a  feasible  and  mandated  solution. 
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Case  Study 


•  Virtual  System  Integration  Lab  (VSIL)  for  the  U.S.  Army 

-  VSIL  is  an  SB  A  toolset  for  accelerating  systems  engineering 

-  VSIL  is  a  flexible  system  integration  suite  for  virtual  prototyping  & 
simulation  that  tests  all  aspects  of  prototype  designs  prior  to  committing  to 
a  physical  prototype 
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VSIL  Objectives 


Enhance  next-generation  vehicle  design 
and  development 

Improve  efficiency  of  simulation 
development 

Perform  cost-benefit  analysis  on 
component  models  up  to  full  deployments 
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Results:  VSIL-enabled  SB  A 


VSIL-enabled 
SBA  Process 


fbJ 
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Repository 


Trend  Analysis 


Simulation 

Execution 

Environment 


Performance 
Analysis  & 
Measurement 


Ptolemy  II 


OpenSIM 


CVS 


GnuPlot 

Excel 


Transforms  development  process  from  one  where  a  vehicle  design  is 
based  on  history  of  direct  ancestors  to  a  process  where  new  vehicle 
design  benefits  from  development  of  all  previous  vehicles 
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Results:  Virtual  Systems  Editing 
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•  Through  the  Virtual  Systems  Editor  (ViSE),  VSIL  provides  an 
integrated  design,  development,  and  simulation  toolset  to  enable 
automated  component  trade  off  analysis  and  requirements  generation. 
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Lessons  Learned  from  VSIL 


SBA  process  adoption  depends  on  the  availability  and  fidelity  of 
component  models  of  interest 

-  (e.g.,  Mobility,  Drive-train,  Suspension,  Mechanical) 

SBA  processes  require  well-populated  component  libraries 

-  The  components  should  be  modeled  at  level  which  they  are  swapped  for 
tradeoff  analysis  (e.g.,  Line  Replaceable  Units) 

Need  model  import  capabilities  for  industry-standard  formats 

-  Flexibility  is  key  for  leveraging  existing  models  and  new  models 

-  Domain  expertise  is  critical  and  an  underestimated  bottleneck 
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Lessons  Learned  from  VSIL 


Powerful  and  user-friendly  tools  make  a  world  of  difference 

-  Increases  the  efficiency  of  Govt  validation 

-  Reduces  the  penalty  on  Industry  for  adopting  SBA 

-  Actual  usage  reduces  anxiety  and  builds  confidence  in  the  tool 
Interface-based  Modeling  &  Simulation  solves  many  IP  issues 

-  Enables  reusability,  gives  Industry  a  layer  of  abstraction  to  protect  its  IP 

Time  savings  can  be  roughly  calculated  by  the  number  of  engineering 
iterations  reduced 

-  The  VSIL  enabled  the  vehicle  systems  engineer  to  do  trade-off  studies 
over  4x  faster  by  reducing  development  iterations 
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Recommendations  for  Effective 
SB  A  Tools  &  Processes 

Populate  centralized  repositories  that  are  government  owned  and 
operated.  But  let  Industry  maintain  proprietary  repositories  with 
interface-based  model  access. 

-  Interface  with  decentralized  repositories  based  on  service  agreements 

-  Provides  Govt  and  support  contractors  real  data  to  use 

Require  the  delivery  of  component  models  developed  under  contract 

-  Govt  &  Industry  need  standardized  tools  to  handoff  and  evaluate  models 

Govt  needs  tools  to  effectively  manage  SB  A  and  M&S 

-  Govt  needs  more  automated  M&S  capabilities 
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Conclusions 


SBA  has  not  failed  -  we  simply  need  more  adoption 

SBA  design  challenges  can  be  overcome  -  but  more  buy-in  and 
mindset  change  required 

SBA  has  many  circular  dependencies  between  Government  &  Industry 
Govt  will  not  get  best  solution  when  the  competition  is  limited 
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Conclusions 


Ultimately,  the  Government’s  degree  of  push  for  SB  A  will  determine 
its  effectiveness.  For  SB  A  to  be  truly  effective  there  needs  to  be  more 
give  and  take  between  government  control  and  industry  free  reign 

Need  more  leadership  from  within  Govt  to  make  this  happen 

War-time  atmosphere  makes  process  change  harder  but  motivating 

More  transformation  is  required,  SBA  is  key  to  Defense  transformation 
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Questions? 

•  Please  address  questions  to  Kevin  Tang,  ktang@  Cybernet .  com 
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Overview 


U.S.AIR  FORCE 


Background  (Helicopter  Brownout) 


Requirements 

■  Mishap  Analysis 

■  Operational  Tasks 


Solutions  Analysis 


Conclusion 


Integrity  -  Service  -  Excellence 


Hovering  Flight 

Not  as  easy  as  it  looks 


Changing  flight  dynamics  during 
approach  to  hover  require  large 
control  inputs 

Flight  Controls 

Main  Rotor  Thrust  Axis 
Main  Rotor  Thrust  Magnitude 
Directional  Control  Inputs  (Anti-Torque) 


\>  All  3  Controls  are 
/  Interdependent 


Flight  Regime 

Static  Instability 
Dynamic  Instability 
Constant  Perturbations 
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Make  Control 
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Pitch  +  Power  t  Aircraft  Control 


The  Question 

How  to  safely  perform  a  vertical  landing 
when  you  can’t  see  outside  due  to 
recirculating  dust/snow? 

Do  something  different  to  keep  visibility 

or 

Replace  and/or  degrade  the  effects  of  lost  information 


Do  Something 
Different  to 
Keep  Visibility 


Land  fast  to  stay  ahead  of  the  dust  cloud 

-  Requires  suitable  long  flat/smooth  LZ 

-  Requires  High  Decel  Rates  at  touchdown 

-  Dependent  on  surface  winds 

-  Aircraft  Specific 

•  Allowable  Landing  Attitude  (Deceleration) 

-  H-60,  H-46,  H-47  Good  (Tail  Wheels) 

-  MH-53M,  Bad  (No  Tail  Wheels) 

-  May  or  may  not  work  out  well  in  formation 


Do  Something  Different  to  Keep  Visibility 


Sometimes  it  works. . . 


Do  Something  Different  to  Keep  Visibility 


Sometimes  it  doesn’t  work 


Replacing  Lost  Information 

(and  degrading  loss  of  capability) 


with  the  aircraft 


you  want  it  to 


. 


Land  Safely  and  Effectively 
Achieve  Combat  Mission  Objectives 


Replacing  Lost  Information 

(and  degrading  loss  of  capability) 


Aircraft  Navigation  Systems 

-  GPS/INS,  Doppler,  Radar  Altimeter 

-  Mission  Computer  (Waypoint  Navigation) 

Low  Speed  Aircraft  Control  Symbology 

-  Drift  Vector,  Vertical  Velocity,  Altitude,  Heading,  etc. 

Geospatial  Information  (What’s  out  there?) 

-  Digital  Map  (Imagery,  Terrain,  etc.) 

-  Sensor  Information  (FUR,  Radar,  etc.) 

Reduced  Aircraft  Control  Workload 

-  Stability  Augmentation 

-  Self  Contained  Approach  Guidance 

-  Coupled  Approach/Hover  Capabilities 


V-22  Hover  Display 


What  can  be  done  to  improve  the  situation? 


Mishap  Analysis 

Long  known  problem,  just  more  prevalent 


33  Identified  USAF  Mishaps  (1971  -2006) 

-  Loss  of  effective  visibility  causal 

-  Landing/Takeoff  phase  of  operations 

-  HH-3E,  MH-53H/J/M,  HH-60G  &  UH-1F 

1980  -  Desert  One  (Operation  Eagle  Claw) 
Failed  rescue  of  American  hostages,  Iran 

Mishap  Costs 

-  $72M  Total  pre  9/1 1  (30  Years) 

-  $72M  Total  post  9/1 1  (5  Years) 

-  DoD  Costs  estimated  at  $100M  per  year 

Mishap  Factors 

-  Inadequate  Aircraft  Control 

-  Undetected  Surface  Hazards 

-  Undetected  Lateral  Obstacle 


CSAR-X  (-$45  Million) 


Mishap  Factor  Inclusion 
33  Total  Mishaps 


100% 


80% 


60% 


40% 


20% 


Aircraft 

Control 


Surface 

Hazards 


Lateral 

Obstacles 


Mishap  Factors  by  Class 

(33  Identified  USAF  Mishaps) 


Class  A  Class  B  Class  C  Class  D  Class  E 


■  A/C  Cntrl  +  Srfc  Hzd  +  Lat 
Obs 

■  Srfc  Hzd  +  Lat  Obs 

■  A/C  Cntrl  +  Lat  Obs 

■  A/C  Cntrl  +  Srfc  Hzd 

■  Lateral  Obstacle 
□  Surface  Hazard 

■  Aircraft  Control 


Mishap  Factor  Inclusion 

5  Fatal  Mishaps 

100% 

80% 

60% 

40% 

20% 

0% 

Control  Hazards  Obstacles 


Conclusions 


•  Maintaining  aircraft  control  is  Problem  #1 

•  Better  awareness  of  hazards  is  important 

•  Reducing  aircraft  control  workload  is  key  if 
added  sensor  information  is  to  be  effective 

•  Capabilities/Tactics  that  allow  zero  roll 
vertical  landings  could  significantly  reduce 
mishaps  due  to  surface  hazards 


Measurable  Requirements 


Used  to  compare  relative  value  between  alternative  solution  sets 
Starting  point  for  tech  demo  system  development 
Framework  for  future  combat  system  development 


Major  Tasks  &  Attributes 


•  Core  Operational  Tasks 

-  Maintain  Geospatial  Awareness  of  Intended  Landing  Point 

-  Confirm  LZ  Size  &  ID/Refine  Landing  Point 

-  Locate  Surface  Hazards 

-  Locate  Proximate  Obstructions 

-  Successfully  fly  to  safe  landing  point  and  land/hover  as  required 

•  Other  Requirements  (System  Attributes) 

-  Human  Factors 

-  Programmatic 

-  Physical  Characteristics 

-  Sustainability 

-  Operating  Environment 


Note:  Related  requirements  sources:  AFSOC  No/Low  Visibility  ICD, 
Cable  Warning/  Obstacle  Avoidance  ICD,  CSAR-X  CDD 


Core  Operational  Tasks  (OV-5) 

Execution  Timeline 


1.  Maintain  Geospatial  Awareness 


3.  Locate  Surface  Hazards 


4.  Locate  Proximate  Obstructions 


5.  Maintain  Aircraft  Control 


Weighted 

Objectives 

Hierarchy 


5  Core 
Tasks 


Performance 


Other  Factors  "" 
(Cost,  Size,  Weight,  etc.) 


Solutions  Analysis 

Process  Overview 


Objective  System  Architecture  (SV-1) 


The  Notional  Baseline  Aircraft 


(MH-53M  +  HH-60G  +  CSAR-X  +  CV-22) 
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INS/GPS 

Turreted  FUR 

Radar  Altimeter 

Digital  World  Model 
(Digital  Map) 

Mission  Computer  /  Flight  Director 

Automatic  Flight  Control  System 
(Waypoint  Nav  &  Coupled  Approach) 

Cockpit  Digital  Displays  (VSD/HSD) 


External  Sensors 

Range,  Resolution,  Penetration 


3D  Capable 

3D  Active  MMW 
LADAR 

Sparsely  Populated  Radar  Array 

IFSAR 

Active  Acoustic  System 
Image  Stereo  Pair  Modeling 


2D  Capable 

IR  Video 

Image  Intensified  (I2  Video) 

Fused  IR/I2  Video 
2D  Active  MMW 

Passive  MMW  (w/  external  illuminator) 

Passive  MMW 

SAR 

Geo-rectified  Digital  Image 


Preloaded  Data 

Digital  Terrain 

Vertical  Obstruction  Data 

Digital  Maps  &  Imagery  (CIB) 


Other  Systems 

Station  Keeping  Equipment  (range/bearing) 
Data  Link  (aircraft  range/bearing  +  sensor  data) 


Primary  Solution  Option 


Technology  Interest  Area 


Aerodynamics  &  Flight  Controls 

Effective  Aircraft  Control 


Manual  Aircraft  Control 

Self  Contained  Approach  Guidance 

Improved  Low  Speed  Stability  (Handling  Qualities) 

Performance  Based  Flight  Controls 

Approach  Guidance  with  Enhanced  Obstacle  Avoidance 


Coupled  Aircraft  Control 

Coupled  Hover 
Coupled  Approach 

Coupled  Approach  with  Enhanced  Obstacle  Avoidance 


Aerodynamics 

Modeling  &  Simulation 
Visible  Null  Areas  (H-101  &  H-53E) 


Primary  Solution  Option 

Research  Area 


Flight  Controls 

No  Addition 

Improved  Aircraft  Handling  Qualities 

Coupled  Approach  with  Enhanced  Obstacle  Avoidance 

Coupled  Approach  with  Enhanced  Obstacle  Avoidance  + 
Improved  Aircraft  Handling  Qualities 


Sensors 

No  Addition 
Sparse  Array 
MMW 
LADAR 


Sparse  Array  +  MMW 
Sparse  Array  +  LADAR 
MMW  +  LADAR 


Sparse  Array  +  MMW  +  LADAR 


Solution 
Configuration 
Matrix  of  1 92 

Human  Interface 

No  Addition 

Helmet  Mounted  Display  (Symbology) 
Head  Tracked  HMD  (Video  &  Symbology) 
3D  Audio  &  Tactile 
HMD  +  3D  Audio/Tactile 
Head  Tracked  HMD  +  3D  Audio/Tactile 


Selecting  Solution  Configurations  for  Evaluation 

Resource/Time  Constrained  (Can’t  Assess  all  192) 


Assess  Acquisition 
Likelihood 

(Cost  Balance,  Synergy) 


Select  Minimum 
Matrix  Sample 

(Design  of  Experiments) 


Select  Additional 
Configurations 
Based  on  Likelihood 


9  (0.1 1)  Baseline  Aircraft  +  No  Additional  Flight  Controls, 
+  No  additional  Crew  Interface  +  No  additional  Sensors 


MZZZii  (0.66)  Baseline  Aircraft  +  Millimeter  Wave 
+  Sparse  Array  Sensors 
+  Head  Mounted  Display  with  Symbology 

+  Improved  Flight  controls  (Handling  qualities) 


Manually  Assessed  as  Likely,  Selected 
Manually  Assessed  as  Possible,  Selected 
Manually  Assessed:  Unlikely,  Not  Selected 
Design  of  Experiment  Selected 


(1 .0)  Baseline  Aircraft  +  Sparse  Array  Sensors 
+  Head  Mounted  Display  with  Audio/ Tactile 
+  Improved  Flight  controls  (Coupled  Approach  with  Obstacle 
Avoidance) 


(0. 11)  Baseline  Aircraft  +  LADAR  Sensor 
+  No  Additional  Flight  Controls, 

+  No  additional  Crew  Interface 


System  Configuration  Assessment 

Generating  Relative  Solution  Value 


Assess  Quantitative 
Component  Measures 

(Sensor  Range,  Weight,  etc.) 


Assess 

System  Measures 

(Quantitative  &  Qualitative) 


Generate  Individual 
&  Composite 
Utility  Values 


<r  o 


0) 

O 

CO  ^ 

t 

CD 

+5  if) 

c  c 

i-2 

E  o 

3 


(0.0)  Baseline  Aircraft  +  No  Additional  Flight  Controls, 
+  No  additional  Crew  Interface  +  No  additional  Sensors 


(4.0)  Baseline  Aircraft  +  Millimeter  Wave 
+  Head  Mounted  Display  with  Symbology 
+  No  Improved  Flight  controls 


(7.25)  Baseline  Aircraft  +  Sparse  Array  Sensors 
+  Head  Mounted  Display  with  Audio/ Tactile 

+  Improved  Flight  controls  (Coupled  Approach  with  Obstacle  Avoidance) 


Sensor  Options  (8) 


Evaluate  Solutions 


Sensors 

-  Millimeter  Wave  System 

•  Low  Task  Performance 

•  High  Cost  &  Size/Weight 

-  Sparse  Array 

•  Moderate  Task  Performance 

•  Low  Cost  &  Size/Weight 

-  LADAR 

•  High  Task  Performance 

•  High  Cost  &  Size/Weight 


m  (0.62)  Baseline  Aircraft  +  No  Additional  Flight  Controls, 
+  No  additional  Crew  Interface  +  No  additional  Sensors 
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Sensor  Options  (8) 


Above  Baseline  (Upper  50%) 
Above  Baseline  (Lower  50%) 
IM  Baseline  (Do  Nothing) 

| 1 ! ' i  i  I  Below  Baseline 


(0.67)  Baseline  Aircraft  +  Sparse  Array  Sensors 
+  LADAR  +  Head-Tracked  Head  Mounted  Display  with  Audio/  Tactile  + 
Improved  Flight  controls  (Handling  qualities  +  Coupled  Approach  with 
Obstacle  Avoidance) 


(0.67)  Baseline  Aircraft  +  Sparse  Array  Sensors 
+  LADAR  +  Head-Tracked  Head  Mounted  Display  with  Enhanced/ 
Synthetic  Vision  +  Improved  Flight  controls  (Handling  qualities  +  Couple^ 
Approach  with  Obstacle  Avoidance) 


(0.48)  Baseline  Aircraft  +  Millimeter  Wave  +  No  Additional 
Flight  Controls, 

+  Head  Mounted  Display  with  Symbology 


Human  Interfaces  •  Flight  Controls 

-  Head  Tracked  HMD  -  High  Benefit 

•  Slim  Benefit  in  High  Cost  Solutions  •  Improved  Handling  Quality  ->  Enhanced  Ops 

•  Slim  Penalty  in  Low  Cost  Solutions  •  Both  >  Either 

-  3D  Audio/Tactile 

•  Penalty  in  Low  Cost  Solutions 

•  Neutral  in  High  Cost  Solutions 

•  High  Cost  for  Stereo  ICS 


Sensitivity  Analysis 

Performance  vs.  Other  Factors  (Cost,  Weight,  Etc.) 


Performance/Other  Solution  Value  Sensitivity 


50-50 


55-45 


60-40 


65-35 


70-30 


Performance  /  Other  Weighting  Ratio 


Helicopter  Brownout  Conclusions 


•  High  Performance  System  (SOF) 

-  Sparse  Radar  Array  +  LADAR  Sensors 

•  Fusion  Processing  &  Persistent  3D  World  Model 

-  Head-tracked  Helmet  Mounted  Display 

•  Symbology 

•  Enhanced  and/or  Synthetic  Vision 

•  Lower  Performance  System  (Conventional) 

-  Sparse  Radar  Array 

-  Helmet  Mounted  Display  (Symbology) 

•  Flight  Control  Systems  &  Guidance 

-  Handling  Qualities 

-  Flight  Directed  &  Coupled  Approach  Capabilities 

-  Assess  and  Develop  per  Aircraft  MDS 

•  Tiltrotor  vs.  Helicopter  Issues 

•  Digital  FCS  vs.  Augmented  Mechanical  Controls 


SE  Wisdom 


•  Clear  measurable  requirements  are  the  most  useful  in 
situations  when  they  are  hardest  to  generate 


•  There  is  nothing  more  potentially  complicated  than  a  blank 
sheet  of  paper 

-  It  is  impossible  to  effectively  consider  and  choose  from  an  infinite 
number  of  design  options  when  developing  a  new  system 

-  Overcome  analysis  paralysis  with  active  management  and  hard 
decisions  to  create  a  manageable  number  of  potential  solutions 


•  Use  systems  engineers  with  actual  ops  experience 
-  Operate  in  the  Region  of  Effective  Communication  (REC) 


Design  Engineer 


Questions? 


Air  Force  Institute  of  Technology 


Integrity  -  Service  -  Excellence 

Systems  Engineering  for  Rapid 
Prototyping:  Friendly  Marking  Device 


NDIA  9th  SE  Conference 

Oct  2006 


Maj  Monte  Cannon 
Maj  Greg  Buckner,  Maj  Greg  Buttram 
Maj  Michael  Jiru,  Maj  Arlene  Collazo 
Dr  Brian  Hermann,  Dr  John  Colombi 
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U.S.AIR  FORCE 


“Can  a  prototyping  development  effort 
be  responsive  enough  to  react  to 
critical  needs  while  still  benefiting 
from  the  rigor  of  systems 
engineering?” 


Integrity  -  Service  -  Excellence 
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Introduction 


U.S.AIR  FORCE 


■  Close  Air  Support  (CAS)  Background 

■  Prototyping  Approach 

■  Friendly  Marking  Device  (FMD)  Results 

■  Conclusion/  Observations 


Integrity  -  Service  -  Excellence 


3 


The  Problem 


U.S.AIR  FORCE 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


■  IAWJP  3-09.3  (2  Sep  05): 

■  Close  air  support  (CAS)  is  air 
action  by  fixed-  and  rotary¬ 
wing  aircraft  against  hostile 
targets  that  are  in  close 
proximity  to  friendly  forces 
and  which  require  detailed 
integration  of  each  air  mission 
with  the  fire  and  movement  of 
those  forces. 


■  Urban  CAS  considerations 

■  Closer  proximity  to  the  enemy 

■  Reduced  communication  time 

■  Presence  of  noncombatants 

■  Potential  for  collateral  damage 

■  Increased  risk  of  fratricide 


Integrity  -  Service 


Background 


Excellence 
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Challenge /  Constraints 


U.S.AIR  FORCE 


■  AF  Research  Lab  Rapid  Reaction  (Core  Process  3) 

■  Compressed  schedule  -  5  months  from  emerging  need  to 
prototypes 

■  No  modifications  to  the  CAS  aircraft  or  pods 

■  Technology  maturity 

■  Resource  availability 

■  Operational  limitations 

■  Cost 

■  Project  Objective:  Develop,  demonstrate  and 
transition  a  marking  solution  that  enables  a  Joint 
Terminal  Attack  Controller  (JT AC)  to  establish  a 
common  point-of-reference  with  a  Close  Air  Support 
(CAS)  asset  such  that  the  CAS  asset  can  attack  an 
intended  target  while  avoiding  fratricide. 


Integrity  -  Service  -  Excellence 


6 


Background 


U.S.AIR  FORCE 


“In  on  going  Close  Air  Support  (CAS) 
missions  and  test  using  MDS  platforms 
with  3rd  Generation  Targeting  Pods;  the 
Joint  Terminal  Attack  Controller  (JTAC) 
working  in  the  Area  of  Objective  has  no 
covert  way  of  friendly  identification.” 

“The  JTAC  needs  a  friendly  marking  device 
that  can  be  seen  by  a  targeting  pod  in 
either  the  FLIR  or  Laser  Spot  Tracker 
mode.  These  emitters  will  increase  the 
pilot  situational  awareness  and  reduce 
fratricide  at  the  same  time.” 
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Rapid  Reaction  Prototyping 


U.S.AIR  FORCE 


Monthl  Month2  Month3  Month4  Month5  Month  5+ 


User 

Workshop 


Concept  Eval 
Matrix 


“DT”  Testing 


Field 
Demo  I 


a 


A 


Continual  updates  of  Utility  Analysis 


TAC-P  CAS  Pilot  Selection 

User  Rgmt  Rqmt  Mtg 
Mtgs 
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U.S.AIR  FORCE 


Ideas 


1 1 1 
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Classic  V-Model 


U.S.AIR  FORCE 


Problem  Definition 


Understand  Requirements 


Derive  Systems  Specs 


Derive  Component  Specs 


Fielded  System 

Demonstrate  and 
Validate  System 


Integrate  and  Verify 
System 

Assemble  and  Verify 
Components 


Derive  Configuration 
Item  Specs 


Fabricate  and  Assemble 
Configuration  Items 


Design  Engineering 


Verify  Configuration 
Items 


Integrity  -  Service  -  Excellence 
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Problem  Definition 


■  Pubs  on  Close  Air  Support  (JP  3-09.3,  Sep  05): 

■  Stakeholder  Interviews  (JTACs  and  CAS  pilots) 

■  User  Requirement  Questions 

■  Analysis  Criteria 

■  Constraints  identification 

■  Restated  problem  as: 

■  The  Joint  Terminal  Attack  Controller  (JTAC)  lacks  a  covert 
means  to  quickly  and  accurately  mark  the  location  of 
friendly  forces  as  a  common  point-of-reference  with  a  Close 
Air  Support  (CAS)  asset  such  that  the  JTAC  can  direct  a 
CAS  attack  with  minimum  risk  of  fratricide. 


Integrity  -  Service  -  Excellence 
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Requirements 
,  Objectives  Hierarchy 


FMD  Prototype 


Prioritize  and 
select  option 

Determine  FOMs 
MOPs  and  MOEs  y 


ndidate  IdentificatioCandidate  Lab 
Candidate  Development 


Range  Tests 


Develop  an  Operational  Concept 


DoDAF  OV-1,  High-Level 
Operational  Concept  Graphic 

DoDAF  OV-5  External 
Systems  Diagram 

Use  Cases  (RUP  template) 


FRIENDLY  MARKING  FOR  URBAN  CLOSE  AIR  SUPPORT 
OV-1:  HIGH-LEVEL  OPERATIONAL  CONCEPT  GRAPHIC 


Success  Guarantee 


CAS  aircraft  provides  bombs  on  target. 


trieide  of  friendly  forces. 


Urban  Close  Air  Support  Use  Case 

Use  Case  Name 

Maine-  Urban  Close  Air  Support 

location  to  die  CAS  aircraft  when  the  target  's  coordinates  are  unknown, 
target  is  a  moving  target,  the  dose  proximity  to  friendly  forces  requires 
higher  degree  of  accuracy,  or  when  time  situation  does  not  allow  for  a 
long  CAS  brief. 

Actors  Involved 

member  who,  from  a  forward  position,  directs  the  action  of  combat 
aircraft  eneaeed  in  dose  air  support  and  other  offensive  air  operations' 

(JP  3-09.3). 

•  Goal  -  To  accurately  identify  CAS  target  and  friendly  position  to 
die  CAS  Aircraft. 

•  To  pres  ent  fratridde. 

Close  Air  Sunnort  :  C'AS1  aircraft  -  Aircraft  tasked  to  sunnort  dose  air 
support  operations. 

•  Goal  —  To  accurately  and  quickly  acquire  CAS  target  and  friendly 
position. 

•  To  prevent  fratricide. 

Air  Summit  Operations  C  enter  (ASOC!  -  "The  principal  air  control 

agency  of  die  theater  air  control  system  responsible  for  die  direction  and 
control  of  air  operations  direcdy  supporting  die  ground  combat  element. 

It  processes  and  coordinates  requests  for  immediate  air  support  and 
coordinates  air  missions  requiring  integration  with  other  supporting  arms 
and  ground  forces”  (JP  3-09.3). 

Preconditions 

•  JTAC  has  constant  communication  with  ASOC. 

•  Target  coordinates  are  not  known,  target  is  a  moving  target  the 
close  proximity  to  friendly  forces  requires  higher  degree  of 
accuracy,  or  when  time  situation  does  not  allow  for  a  long  CAS 
brief. 

•  JTAC  contacts  the  ASOC  to  request  CAS 

•  ASOC  contacts  CAS  aircraft  and  passes  them  CAS  information 
(coordinates,  frequencies,  JTAC  info,  etc.) 

•  CAS  Aircraft  contacts  JTAC 

ption  to  CAS 
gpodtoIR 

D)to 

us  mode. 

ft  using 

ircraft  notifies 
i. 

and  realiens  it 

;tme  pod  in 

onal  10  nm. 
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Problem  Definition 

\Operational  Concept 


^  Objectives  Hierarchy 
Definition  of 
system  level 

■  MOFs  ft  MOPs 


FMD  Prototype 

Prioritize  and 
select  option 

Determine  FOMs  wjf 
MOPs  and  MOEs  > 

Range  Tests 


candidate  IdentificatioCandidate  Lab  Testy 
Candidate  Development 


Requirements  Analysis 


Use  Case  refinement 

User  Requirements  with  weights 

■  JTACs 

■  CAS  Pilots 
FURPS+  model 

■  Functional 

■  Usability 

■  Reliability 

■  Performance 

■  Supportability 

■  “plus”  other  requirements  such  as 
Implementation,  Interface, 
Operations,  Packaging,  Legal,  etc. 


User  requirements  with  weights 

Types  of  Requirements 

Requirements 

Environmental 

Weather  Limitations 

Day/Night  Limitations 

Physical 

Waterproof 

Shockproof 

Power  Source 

Weight 

Size  Dimensions 

Operational  (signal) 

Signal  Duration 

Signal  Covertness 

Signal  Field  of  View 

Signal  Range 

Accuracy  Resolution 

Signal  Spectrum 

System  Compromise 

Unique  Signal 

Signal  Transmission  Delays 

Operational  (system) 

Ease  of  use  /  training  required 

Modification  required 

Unique  Signal  display 

Acquisition  (Long  term) 

Long-term  unit  cost 

Product  Feasibility 

Acquisition  (Short  term) 

Estimated  cost 

Prototype  availability 

System  Maturity/  estimated  TRL 

Factors  influencing  prototype  development 

Integrity  -  Service  -  Excellence 
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Problem  Definition 


FMD  Prototype 


Objectives  Hierarchy 


User  defined 


Mark  Position 


Environmental  |  | 

Element  Value 

0.9 

0 

Welther 

0.8 

0 

Daf/Nigi 

Score  0 

Physical 


0.9 

0.8 


Element  Value 

0 

Waterpn 

0 

fehockpri 

0  i 

[Power  S 

0  / 

Weight 

o  / 

Size 

Element  Valle 


Range  Utility  Curve 


5  NM 

Threshold 

0  1  2  3  4! 

i  6  7  8  9  10  11 

-  Series  1 


Range  (NM) 


Risk-Neutral  Utility  curves 

Based  upon  user  requirements  & 
expressed  desires 

Each  candidate  is  scored 

Updated  as  candidates  matured  (ie: 
test  data) 

Long/short  term  acquisition  elements 
based  upon  engineering  judgement 


Weight 

Score 

Ease  of  Use 

Modification  Rqd 

Jnique  Signal  Display 

Weight 

Score 

.ong  Term  Unit  Cost 

Production  Feasiblity 

Weight 

Score 

Estimated  Cost 

Prototype  Avail 

■RL 

Integrity  -  Service  -  Excellence 
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Problem  Definition 


FMD  Prototype 


Monthl 


Month2 


Objectives  Hierarchy:  “Living  Tool” 


Month3  Month4  Month5  Month  5+ 


T 


First  Down  Select 


Component  Tests 


Transition 

Go-No  Go  Select  Recommendation 


Integrity  -  Service  -  Excellence 
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Problem  Definition 


FMD  Prototype 


Definition  of  system  level 

MOEs  and  MOPs 


m  be  eatable  ©fbei 


Originating  Requirements 

■  Weight :  4-  6  oz  without  batteries 

■  Volume  :  25. 1  in3 ,  “Less  than  a  Coke  can  “ 

Critical  operational  issues  (COI) 

■  The  JTAC  carries  a  variety  of  mission  equipment  to  execute  a 
mission.  The  JTAC  has  limited  excess  space  and  weight  capacity 
for  carrying  new  mission  equipment. 

Measures  of  effectiveness  (MOE) 

■  Solution  shall  be  capable  of  being  carried  by  a  JTAC  outfitted  with 
a  typical  complement  of  mission  equipment. 

Measures  of  performance  (MOP) 

■  Weight  of  the  solution  including  packaging  and  expendables. 

■  Volume  of  the  solution  including  packaging  and  expendables. 


Integrity  -  Service  -  Excellence 
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Problem  Definition 


FMD  Prototype 


Identify /  Develop  Technology 

Candidates 


■  AF  Research  Lab  (AFRL)  already 
had  many  concept  ideas 

■  Team  utilized  several  “ brain 
storming ”  sessions  to  refine 
possible  technologies 


1064  Laser 

1064  Laser 

1064  Laser 

1064  Laser 

1064  Laser 

3-5  micron  LED 

3-5  micron  LED 

3-5  micron  LED 

3-5  micron  LED 

3-5  micron  LED 

3-5  micron  LED  Beacon 

3-5  micron  LED  Beacon 

3-5  micron  LED  Beacon 

3-5  micron  LED  Beacon 

3-5  micron  LED  Beacon 

873  Laser 

873  Laser 

873  Laser 

873  Laser 

Box  Thermal  Emitter  Arrav 

Box  Thermal  Emitter  Array 

Box  Thermal  Emitter  Arrav 

Box  Thermal  Emitter  Arrav 

Haloqen  Bulb  Flashlight 

Haloqen  Bulb  Flashlight 

Halogen  Bulb  Flashlight 

Halogen  Bulb  Flashlight 

Halogen  Bulb  Flashlight 

Israeli  COTS  Emitter 

Israeli  COTS  Emitter 

Krypton  Bulb  Flashlight 

Krypton  Bulb  Flashlight 

Krypton  Bulb  Flashlight  1  Krypton  Bulb  Flashlight 

Krypton  Bulb  Flashlight 

Laser  Beeper 

Laser  Beeper 

Laser  Marker 

Laser  Marker 

Laser  Warning  Receiver 

Laser  Warni  ng  Receiver  I  Laser  Warni  ng  Receiver 

Laser  Warning  Receiver 

Rescue  Laser  Flare 

Rescue  Laser  Flare 

RF  Tag 

RFTag 

Special  Material  Locator 

Special  Material  Locator 

Special  Material  Locator  I  Special  Material  Locator  I  Special  Material  Locator 

TRON1 

TRON  1 

Thermal  Beacon 

Thermal  Beacon 

1 

Thermal  Signaling  Device  1 

Thermal  Signaling  Device  1 

Thermal  Signaling  Device  1  iThermal  Signaling  Device  1 

TRON  2  (TBD) 

TRON  2  (TBD) 

TRON  3  (Blanket) 

TRON  3  (Blanket) 

TSDII  (Helio  Triad) 

TSD  II  (Helio  Triad) 

TSD  III  (Cal  Triad) 

TSD  III  (Cal  Triad) 

TSD IV  (Single  Space  Heater) 

TSD  IV  (Single  Space  Heater) 

TSD  V  (Space  Heater  Triad) 

TSD  V  (Space  Heater  Triad) 

■  m 


Thermal  Emitter 

*  e 

Box  Array 

*  A  ,  .sff  dS  dS 

e  ##'  *  m 

. -  -■  . 


Laser 


Integrity  -  Service  -  Excellence 
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FMD  Prototype 


Problem  Definition 


Candidate  Lab  Tests 


■  Component  level  testing  conducted  during  prototype 
development 

■  Integration  of  all  the  pieces 


Integrity  -  Service  -  Excellence 
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Range  Test 
Go/No-Go  Selection 


■  Prototype  Testing  &  Production  Estimates 

■  Confirming  pre  test  mathematical  analysis 

■  Component  test  results  -  Detection  Range 

■  Objective  Hierarchy  updates 

■  Final  Go  /  No-Go  Selection 


Problem  Definition 

\Operational  Concept 


FMH  Prntnt\/np> 


Requirements 
^  Objectives  Hierarchy 

Definition  of 
system  level 

■  MOF.s  ft  MOPs 


andidate  IdentificatioCandidate  Lab  Test: 


Candidate  Development 


Integrity  -  Service  -  Excellence 
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FMD  Prototype 


Problem  Definition 

\Operational  Concept 


Requirements 
^  Objectives  Hierarchy 

Definition  of 
system  level 

■  MOF.s  ft  MOPs 


andidate  IdentificatioCandidate  Lab  Test: 


Candidate  Development 


■  Development  of  Prototype  Test  Plan 

■  Prioritized  Test  Point  Matrix 

■  Highest  weighted  areas  in  Objective  H 

■  Objectives 

■  Determine  Detection  Range 

■  Operator  Usability  Assessment 

■  Flight  Profiles 

■  Profile  1  -  Open,  flat  terrain 

■  Profile  2  -  Urban  complex 

■  Profile  3-  Elevated  terrain,  stand-  off  pos 

■  Evaluation 

■  Sniper  &  LITENING  pods  " 

■  F-15E,  F-16,  A-10  aircraft  mix 


Integrity  -  Service 


Range  Test  Plan 


-  Excellence 
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FMD  Prototype 


Problem  Definition 

\Operational  Concept 


Requirements 
^  Objectives  Hierarchy 

Definition  of 
system  level 

■  MOF.s  ft  MOPs 


andidate  IdentificatioCandidate  Lab  Test: 


Candidate  Development 


Example  Test  Setup 


Integrity  -  Service  -  Excellence 
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Problem  Definition 


FMH  Prntnt\/np> 


Range  Test  (A-10  at  llnm) 


Integrity  -  Service  -  Excellence 
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Problem  Definition 

\Operational  Concept 


FMH  Prntnt\/np> 


Requirements 
^  Objectives  Hierarchy 

Definition  of 
system  level 

■  MOF.s  &  MOPs 


Summary  Test  Results 


^ndidate  IdentificatioCandidate  Lab  Testy 
Candidate  Development 


TEB  &  TSD  V  longest  detection  range 

Aircrew  assessment 

■  Pod  Narrow  Field  of  View  -  best 

■  Modulated  signal  easier  to  pick  out 

■  Current  configurations  good  for 
convoy  support  now 

JTAC  assessment 

■  Detection  ranges  exceed  expectation 

■  Instant  turn  on  and  off 

■  Hands  free  operation  preferable 

■  NVG  Covert  still  nice  to  have 

■  Multiple  modulation  rates 


Device 

F-15E  Sniper 

Predator 

A-10  -  LITENING 

TEB  (20) 

12  nm 

9.5  nm 

18  nm 

TEB  (12) 

6  nm 

10  nm 

not  tested 

TSD  II 

4  nm 

11  nm 

11  nm 

TSD  III 

3  nm 

12  nm 

11  nm 

TSD  IV 

11  nm 

11.5  nm 

10  nm 

TSD  V 

not  tested 

10  nm 

18  nm 

LED 

no  detection 

no  detection 

not  tested 

Israeli 

not  tested 

no  detection 

not  tested 

LWR 

not  tested 

not  tested 

dead  battery 

Integrity  -  Service  -  Excellence 
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Problem  Definition 

\Operational  Concept 


Requirements 
^  Objectives  Hierarchy 

Definition  of 
system  level 

■  MOF.s  &  MOPs 


Prioritize  and  Select  Options 


andidate  IdentificatioCandidate  Lab  Test: 


Candidate  Development 


Thermal  Emitter  Box 

-  Detection  distance  greater  than  10  nm 

-  Potential  to  miniaturize  for  helmet 
mounting  (hands-free) 


Thermal  Emitter  Box  Array 


Special  Material  Locator  Marker 


Thermal  Emitter  Beacon  (Box  array) 

0.86 

Special  Material  Locator  Marker 

0.82 

Thermal  Signalling  Device  II 

0.65 

Thermal  Signalling  Device  III 

0.65 

Thermal  Signalling  Device  IV 

0.60 

Thermal  Signalling  Device  V 

0.60 

LED  (3-5  mic) 

0.47 

Integrity  -  Service  -  Excellence 
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Conclusion 


U.S.AIR  FORCE 


■  Application  of  systems  engineering  rigor 
compatible  under  “rapid  response” 

■  Technology  available  to  identify  friendly  forces 
during  urban  CAS 

■  Several  SE  Observations 

■  SE  can  be  tailored  to  rapid  prototyping  while  maintaining  rigor 

■  Understanding  key  constraints  and  the  larger  context  provided  a 
decision-making  framework  for  the  project 

■  Proven  techniques  from  software  engineering  were  applicable  in 
a  rapid  hardware  prototyping  effort 

■  Selection  of  SE  tools  facilitated  the  decision-making  process 

■  The  systems  engineering  team  helped  link  users  and  technology 
providers  together  to  produce  an  effective  collaboration 

■  Parallel  COTS  Integration  reduced  overall  risk  of  the  project 

■  Priority  given  to  the  project  varied  across  participants 

■  Rapid  prototyping  requires  a  creative  transition  plan 


Integrity  -  Service  -  Excellence 
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Air  Force  Institute  of  Technology 

Integrity  -  Service  -  Excellence 

?  QUESTIONS  ? 


Dr  John  Colombi 
john.colombi@afit.edu 
AF  Center  for  Systems  Engineering 
937-255-3355  x3347 
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Introduction  to  GEIA-STD-0007 
Logistics  Product  Data 

James  Colson 
U.  S.  Army 

Logistics  Support  Activity  (LOGSA) 

U.S.  Army  Materiel  Command 
Logistics  Support  Activity 
Sparkman  Center ,  Bldg  5307 
Redstone  Arsenal ,  AL  35898-7466 

www.locisa.armv.mil 


Oct  2006 


Outline 


Introduction  -  Why  GEIA-STD-0007? 

■  Content 

■  Supportability  Analysis 
Relationships 

■  Supportability  Products/ 

Contracting 

■  Relationship  to  AP239 

■  Summary/Schedule 


October  2006 
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Acquisition  Reform 


OSD  Mandate  For  Change 
-  Dr.  Perry’s  Guidance  June  1994 

DOD  Will  Rely  on  Commercial  Products  and  Processes 


MIL-STD-1388  STDs  Cancelled  -  1996 

■  Ten  Years  Since  MIL-STD-1388-2B  was  Eliminated 
-  What  are  program  offices  using? 


October  2006 
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LSAR  Utilization  Survey 


Survey  Results  Of  Army  Major  Weapon  Systems 


■ 

Number  Of 
Responses 

Number 

With 

Logistics 

Data 

Delivery 

Rqmts 

Number 
Using  LMI 
Spec 

Number 
Using  LSAR 
Standard 

Number 

Using 

LSAR 

System  For 
Data 
Delivery 

ACATI 

23 

19 

13 

5 

19 

ACAT  II 

14 

10 

4 

4 

6 

TOTAL 

37 

29 

17 

9 

25 

Note:  25  of  29  (86%)  Contractors  Provided  data  in  1388/LSAR  format! 


October  2006 


Where  are  we  Going? 


Fact:  Re-establishing  MIL-STD-1 388,  Will  NOT  happen! 


Direction  Industry  Standards 

Utilizing  the  Defense  Acquisition  Life  Cycle  Framework 

-  Effectively  Replacing  MIL-STD-1 388-1 A  LSA  Processes 

Working  within  the  Framework  of  the  Government  Electronics  and 
Information  Technology  Association  (GEIA) 

-  GEIA-STD-0007,  Logistics  Product  Data 

-  GEIA-HB-0007,  Handbook  and  Guide  for  Logistics  Product  Data 

-  Effectively  Replacing  MIL-STD-1 388-2B,  LSAR  Data  Exchange 


October  2006 


SiBJG  Picture” 


DoD  Framework 


Supportability 

Analysis 


AP239  Activities 


System  Support 


IETM 
Training 
Provisioning 
Parts  Lists 
Etc... 


October  2006 
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DOD  Life  Cycle  Management  Framework 


Defense 

Acquisition 

System 

■Iflvsii  L  driven} 


Planning, 

Programming, 

Budgeting, 

&  Execution 

Process 


Joint 

Capabilities 
Integration  & 
Development 
System 

(iwl  driven) 


Integrated  Defense  Acquisition,  Technology,  &  Logistics  Life  Cycle  Management  Framework 


rbp  M/testone  Decision  A  uttiwity  way  authorize  entry  Into  ( Jw  ,T«ju(lFrf*'iT  proem  af  any  point.  consistent  with  phase-speerlk  entrance  criteria  end  statutory  requirements' 


Concept  Rel'memeut  Phase  - 


Production  4  Deplayrnenl  Phase  - 


-»  Operfllions  &  Support  Phase 

ro Jw  Kuf  It* -ry-rt*.  i 


iHI-Hale  PrwUMitlonrPeptovmcrU  Stitlalnmonl  Diapoagl 


Oversight  & 
Review 


Contracting 


Major 

Products 


October  2006 
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Life  Cycle  Management  Framework 


Prototypes/ 
Engineering 
Dev  Models 


3 


Product  Support  Plan 

■SE^hJtoFV-'Eegulalorv  -Product  Support  Element* 
■Sourceot  Support  '  -i-LtolY  -E-uppiTl  -Trdrdrpq 
■Leqacv  Coneideralion^  -l«l™wrom5  -SuppenDda 
?  p  -Ntifipofl4r-&p4r*arfi4l 


Prototypes/ 
Engineering 
Dev  Models 


Initial 

Production 

Baseline 


Demonstrate  Product  Support  Capability 
■Footprint  Reduction 
-S i_p p I v  Chain  Management 
■Froducl  Support  Elements 


INPUTS 


•Syc  Panama  no  a  Spec 
■Eje'r  Orta"' a 
"Vai'if  aivrf  Syc  Supp  art  £ 
Mainrenance  Cope ru#a  £ 
ftATUiVim  anra 
-.4P0 
■COO 
■SEP 
■ISP 
■TEMP 


_ \  DRR 

Performance  Basod  Logistics  (PBLj  Strategy  .^mtjrtDDK>oa:isi-/pi>n^p!F^ 

Produc  I  Supperl  kilegralor' 
Pro-dirt  Support  Fre/kbr 


Pc-H-H-rivri*:-:*  Bated 
Agree  me  rle 


Bu-ilneea  Case 
Aralytk 


■’jr  ■v 


Interpret  User  Heeds-. 

Flefine  Sysrlem 
PerfoHiftonce  Cpece  4 
Emir  on  mental  Conalnoribe 


Logietiee  &  Techrioal  Acronym* 


>  By  ■;  l 

Lew  Bab  killalProdutlen 


ASR  -  Albindlw  Err  den-  Revbw 
BLFiP-  Bey-md  Le 
■I  DR-  CrlkalBeelgi  Revba' 

01-  Gonlgiralkxi  lem 

DTf.E  -  Developments  Teel  £  Evduallon 

EOA-  Eflrly  Operalenal  Aeeeeemenl 

ESOH  -  ErwIrermeriaL  SdeJy  f.  Oik  ipdlenal  HeaHh 

FCA  -  Fmcllenal  Ccrtlqirallen  Audi 

FHEGA  -  Failure  Mode  EtfectH  4  Crilk  Jllv  AnatAb 

FOT4E  -Fdlowon  Teel  f.  Ev-jNj Jl-:ri 

FTA  -  Failure  Tree  Andys  k 

IOTEE  -  Inlld  Cperalkral  Teel  &  Evalualkxi 

I&R  -  In-Servke  Hevbw 

I P  -  kilormdlen  £upporl  Han 

ITH-  hllblTe*ftrfcalRe7le* 

jrTD-  Jotrt  bier  opera  billy  Teel  Cemmand 

LR6E  -  Lbe  Ire  Teel  4  Ev-Ju 

LORA-  Level  el  Fb  pair  Analyse 

LR1P-  Low  Rab  UMLiI  PiwIlk  lion 


OTf. 

OTRR- 


O  cere  I  krai  Teel  A  Evduallcn 
:-p  ere  I  kra  I  Teel  Fbadneee  Fbvbw 


PESHE  -  Progranmdk  Erwlronnenl. 

Sably4  Occ  upalkral  Hedlh  Evduiilon 
POH  -  Preliminary  Deelgn  Fbvbw 
PCA  -  Ptiy;  k  J  Cenllguralbn  Audi 
PRR  -  Predudlon  Reodneee  Revbw 
PPP  -  Progium  Pw«  itri  PI  on 
F?IH  -  F;  el  h  billy  Cenbred  Hdnbrunce 
RMS  -  FielkiMliy.  HalriahJ^lly*. 

Supperiiddlly 

SEP  -  ^-yilemi'  Ercjneerkig  Pijn 
SFR-  Sytlem  Funcltond  Revbw 
SRR  -  S-yelen  Requfre  merle  Revle* 
STA- Syelem  Threal  Aeeeeemert 
BWH  -  Syelem  Verilkallon  Revbw 
Tf.E  -  Ted  b  Ev-jNj  Jlen 
TEMP  -  Teel  4  Ev dm Lk*i  Maeler  Pkm 
TDS-  Technology  Develepmeri  Strabgy 


OUTPUTS 

ym'  -In  #iai  Pfod  daeeiVue 
■T eer  h^ftv 
*TEMP 

■E  lenienre  a  t  Pw  duct 
Suppe  ft 

•Rick  Acccccm  an  t 
■SEP  rTRA 
■PEStfE 
•in put*  to: 

-CPQ  -STA  4SP 
-Cgg^lufanftfHi-af  Eat. 


C 


Combined  DT4  E'OT-fcE'LFTftE 
Demonetrole  S-.ietemto 
Specified  User  Meede  & 

En  dronnwnlal  Conetrainta 


Dv-*Jop  See  tern 
Fundionol  Spoce  S, 

I  Syeteni  Vori  Ficalion  Plan 

OAV^p« Jl-:tid  Aeeeeemert 

THR-Teel  Readheee  Revb  A 

Sy*tm  DUE,  LFTiE  y.  0A*, 

VerHySye-tm  Fu notions Ete' 

4  Cont'troinkr  Compl^noe 
to  Spoo* 

K-  ..............  ......... - - - - - - - 

CTRR3  £,  ^ 

JF  M  ECA^. 

[rr.J 


J_ORj3 


MTA  . 


Evd¥o  C 

^Functional 

Specs  ir 

11  o  Product 

(ij.il  loi  i 

un>enlation 

and  \vi/\ \ 

section  Pbn 

Ev^ok'e  Functional  \ 
F^riormgnco  Spece  into  \ 

Integrated  DT4E.  LFT-feE  & 

FUa-b  V iii k i k1  P^friRTvirvfl 

Cl  Functional  iDeeigntojl 
Spoce  and  Cl  Veri  Hcotion  Plan 

t ■—  .“.r  1  tTIIi  rvrivm  rJI  P.  nr 

Complianceto  Spece 

c^: 


Indk  iducJ  Cl 
Verification 

DT-feE 

CDRj 

Fabricate.  Aeaemble. 

Code  to  ::Buld-to" 
Documentation 

October  2006 


GEIA-S  JJD-,L)00/ 


Define  Logistics  Product  Data  Generated  During 
Design  of  a  System,  End  Item,  or  Product 


Define  Data  Exchange  Mechanisms 


■Legacy  Co  ri  eider  aliens- 


ASR- 


Prototypes/ 
Engineering 
Dev  Models 


Product  Support  Plan 

■Stotutorv-'F'e-quIatcrv  ■Product  Support  Elements 

suPPort;  -P^Da 

r  pens  rri  eJ 


Prototypes/ 
Engineering 
Dev  Models 


Initial 

Production 

Bas-eline 


Demonstrate  Product  Support  Capability 

■Footprint  Reduction 
■SLpply  Chain  Management 

■Product  Support  Elements 

_ ) 

DRR 


Performance  Based  Logistics  (PBL]  Strategy pMjrriPmiirtsufpvAApfr'. 

Peri>:rrran>:e  E>used  &-ij;lrpe;-;  Ciw  Produc  I  Support  hlegraLm' 

"  M  "  '  -mSvi 


Agree  merts 


A  tl'i  I I 


Produd  3-uppcrl  Provider 


4 

"*>r  ‘"V 


c 

-\ 


Logietice  S.  Technical  Acronyms 


-  Albmdlve  'E-vsIer*  Rv^w 

V  Rale  fe-illul  Produc  Ion 
CDR-  Crlloal  Deelgi  Review 
-  Ccnigwallcn  l"e  m 
DTf.E  -  Development  TmI  4  Evoluallon 
EOA-  E-'rlv  Operalerul  As  ses  smom 
ESOH  -  ErwIrormoniaL  S-Jory  4  O-:-:  '.p-ilonal  Healih 
FCA  -  Fincllonal  Ccrflqirallon  Audi 
FHECA  -  Failure  Mode  llteote  4  Ortlkulty  An affile 
FOT&E  -Fmic^sn  Teel  4  Euabdlon 
FTA  -  Follire  Tree  And1, vis 
lOTfcE  -  Inftld  Operational  Teel  EL  Evaluation 
IE-R  -  In-Servke  Hevlew 
IBP-  hlormdlon  £*upp«rl  Plan 
ITH-  hlllal  Tech  rial  Rev  lew 
JITC-  JoJrt  hleroperabllN  Teel  Command 
LFT&E  -  Lh/e  nre  Te.-l  4  Ev-JuaiLin 
LORA-  Level  el  Repair  Aralyste 
Uhl  Produc  lion 


LR1P-  Lott  Fab  kill 


OT4E  -  Operational  Tei  I  &  EvcduaiLin 
OTRR  -  ■: feral trtjl  T eel  Reodneesi  Review 
PE  S4H  E  -  Prog  ran  nulls  Envlrcnnenl. 

Salely 4  Oct  up-'iiL:n'il  Hedih  Evaluation 
POH  -  f'Ktlr  *v  Deelqn  Review 
PC  A  -  Phyek  id  0  enikfirallen  Audi: 
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Exchan  g  e  M  echanisms 


Standard  DEX  1 


•  Central  Exchange 

Federated  database,  must 
establish  exchange 
agreements  with  each 
“Partner”. 

•  Point  to  Point 

Simple  exchange,  must 
establish  exchange 
agreements  and  Protocols 
with  each  Partner 


•  Closely  Tied 

Related  exchange,  typically 
similar  data  and  structure 
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1 GEIA-STD-0007  Content 


Functional  Area  Entities/Attributes 

-  Cross  Functional  Requirements 

-  Operations  and  Maintenance 

-  Reliability  Requirements  and  Analysis 

-  Task  Analysis 

-  Skill  and  Training 

-  Support  Equipment 

-  Unit  Under  Test 

-  Facility 

-  Transportability 

-  Provisioning  and  Cataloging  Requirements 
■  Data  Types  Dictionary 

XML  Schema  for  Logistics  Product  Data 

-  Update/Change  Process 

XML  Schemas  for  Transaction  Sets 


-  Provisioning  &  Style  Sheet 

-  Packaging  &  Style  Sheet 

-  Task  Analysis 

LCN,  ALC,  UOC  Assignment  Guidance 


October  2006 


GEIA-STD-0007  Data  Model  Example 


H  G_part_application_ 

H  AJtemJ  d  entifi  catio  n_ 

p  ro  visio  n  i  n  g_d  ata 

- ! - U - 

data 

Cl_Task_P  revision  ed_ 
Item  Data 


XB_logistic_support_analysis 
_con  tro  l_n  umberj  nde  ntured 
item  data 


CG_Task_Support_ 

Equipment_Data 


XA_endJtem_acronym_c 
ode  data 


CA  Task  Requirement 
Data 


CD_Subtask  Personnel. 
Requi  remen  t_Data 


C  K_Ta  s  k_l  n  ventory 
Data 


CH_task_ 

manual 


CB_Subtask_ 
-o  Requirement 
Data 


XMechnica  l_ma  nu  a  Icode 
and  number  index  data 
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GEIA-STD-0007  Data  Element  Dictionary  Example 


ELEMENT  TYPE 


TYPE 


MAX 

LENGTH 


DEFINITION 


achieved  _availability  _Type  decimal  9  The  probability  that,  when  used  under  stated  conditions  in 

(0-2)N[.(0-6)N]  an  ideal  support  environment,  a  system  will  operate  satisfactorily 

at  any  time.  This  differs  from  Inherent  Availability  only  in  its 
inclusion  of  consideration  for  preventive  action.  Aa  excludes 
supply  downtime  and  administrative  downtime.  The  measurement 
bases 

for  MTBM  and  M  must  be  consistent  when  calculating  Aa. 

Aa  may  be  expressed  by  the  following  formula: 

Aa  =  MTBM 

MTBM  +  M 

where 

MTBM  =  (_L _  1  1  )  - 1 

MTBF+  MTBM-ND  +  MTBPM 
N 

E  (ETi)  (TFi) 
i=l _ 

M  =  N 

E  TFi 
i=l 

M  =  Mean  active  maintenance  downtime  (where  corrective  and 
preventive 

actions  are  considered)  ETi  =  Elapsed  time  for  task  i 
TFi  =  Task  frequency  for  task  i 
N  =  Total  number  of  tasks  performed 

Note:  The  measurement  bases  for  MTBF,  MTBM-ND,  and  MTBPM 
must  be  consistent  when  calculating  the  MTBM  parameter. 
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GEIA-STD-0007  XML  Schema  Example 


<?xml  version="1.0"  encoding="UTF-8"  ?> 

<!—  xmlns:geia  is  needed  by  xpath  in  key/keyref  as  xpath  does  not  work  with  default  namespace  — > 

_  <xs:  schema 

xmlns:geia="http://www.geia_STD_0007.com/2006/schema" 
xmlns:lsartypes="http://www.geia_STD_0007.com/2006/types" 
xmlns:xs="http://www.w3.org/2001/XMLSchema" 
targetNamespace="http://www.geia_STD_0007.com/2006/schema" 
elementFormDefault="qualified"  attributeFormDefault="unqualified"> 

<xs:  import 

namespace="http://www.geia_STD_0007.com/2006/types" 
schemaLocation="GEIA_STD_0007_Types.xsd"  /> 

_  <xs:complexType  name="XA_end_item_acronym_code_data_type"> 

_  <xs:all> 

<xs:element  name="end_item_acronym_code" 

type="lsartypes:end_item_acronym_code_Type"  /> 

<xs:  element  name="logistic_support_analysis_control_number_structure" 

type="lsartypes:logistic_support_analysis_control_number_structure_Type"  minOccurs="0"  /> 
<xs:element  name="administrative_lead_time" 

type="lsartypes:administrative_lead_time_Type"  minOccurs="0"  /> 

<xs:element  name="contract_team_delay_time" 

type="lsartypes:contact_team_delay_time_Type"  minOccurs="0"  /> 

<xs :  element  name="contract_number " 

type="lsartypes:contract_number_Type"  minOccurs="0"  /> 

<xs:element  name="cost_per_reorder_action“ 

type="lsartypes:cost_per_reorder_action_Type"  minOccurs="0"  /> 
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GEIA-H 


07  Handbook 


■  GEIA-HB-0007  (The  Handbook!) 
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GEIA-H  B-0007  Outline 


Overview  of  how  (e.g.  what  analysis)  and  when  Logistics  Product 
Data  is  generated  during  the  development  process  (AP239  and 
DOD  Lifecycle  Models) 

Description  of  the  Logistics  Product  Data  Entities  and  Attributes 
(When  Required,  Sources,  Indenture  Level  Relationships,  Primary 
Use) 

■  Contracting  for  Logistics  Product  Data 

■  Appendices 

-  Sample  Relational  Tables 

-  Test  Data  Set 

-  LCN,  ALC  and  UOC  Guidance  to  Include  Relationship  to  S1000D  SNS 

-  Data  Cross  Reference  List  (0007,  DEF  STAN  00  60,  MIL-STD-1388-2B) 

Publish  Dec  06 


October  2006 


16 


Supportability  Analysis  Process 


Concept  Refinement  Phase/Generate  Support  Solution 


POD  Life  Cycle  Framework  Analyses 


•Perform  Use  Study 
•Perform  Comparative  Analysis 
•Identify  Standardization 
Opportunities 
•Functional  Requirements 
Analysis 

•Define  Supportability  Factors 


AP239,  Product  Life  Cycle  Support  Activities 


•Define  Support  Context 
•Life  &  Usage  Profile 
•Available  Resources 
•Establish  Requirements 

•Elicit  Stakeholder  Needs 
•Define  Support 
Requirements 


GEIA-STD-0007 


•X  Entities  -  Cross  Functional 
Requirements 
•A  Entities  -  Operations  & 
Maintenance  Requirements 
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Supportability  Analysis  Process 


Technology  Development  Phase/Generate  Support  Solution 


POD  Life  Cycle  Framework  Analyses 


•Update  Comparative  Analysis 
•Identify  Standardization 
Requirements 

•  Define  Functional  Requirements 

•  Conduct  Tradeoff  Analysis 
•Conduct  Sensitivity  Analysis 
•Conduct  Limited  Task  Analysis 


AP239,  Product  Life  Cycle  Support  Activities 


•Define  Support  Solution 

•Establish  Support  Drivers 
•Task  Analysis  -  Potential 
Tasks 

•Predict  Support  Performance 
&  Resource  Use 


GEIA-STD-0007 


•X  Entities  -  Cross  Functional 
Requirements 
•B  Entities  -  Reliability 

Requirements  &  Analysis 
•C  Entities  -  Task  Analysis 
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Supportability  Analysis  SProosss 


System  Development  &  Demonstration  Phase/Generate  Support  Solution 


AP239,  Product  Life  Cycle  Support  Activities 


•Predict  Support  Performance 
&  Resource  Use 
•Task  Analysis 
•Define  Support  Solution 
•Support  Policy 
•Support  Plan 
•Task  Procedures 
•Assemble  Solution 
•Assess  Support  Performance 


GEIA-STD-0007 


•X  Entities  -  Cross  Functional 
Requirements 
•B  Entities  -  Reliability 

Requirements  &  Analysis 
•C  Entities  -  Task  Analysis 
•E  Entities  -  Support  Equipment 
•U  Entities  -  Unit  Under  Test 
•F  Entities  -  Facilities 
•G  Entities  -  Skills  &  Training 
•H  Entities  -  Provisioning  & 
Cataloging 


5U 


POD  Life  Cycle  Framework  Analyses 


•  Define  Functional  Requirements 
•FMECA 

•Failure  Tree  Analysis 
•RCM 

•Task  Analysis 
•LORA 

•Supportability  Testing 


Supportability  Analysis  Process 


Production  &  Deployment  Phase/Commission  Support  Solution 


POD  Life  Cycle  Framework  Analyses 


‘Supportability  Testing 
•Provisioning 

•Provisioning  Screening  (Cataloging 
•Early  Fielding  Analysis 


AP239,  Product  Life  Cycle  Support  Activities 


•Assess  Support  Performance 
•Define  Support  Solution 
•Provisioning 


GEIA-STD-0007 


•X  Entities  -  Cross  Functional 
Requirements 
•H  Entities  -  Provisioning  & 
Cataloging 

•All  Other  Entities  Affected  by 
Testing 
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Supportability  Analysis  Process 


Operation  &  Support  Phase/Provide  Support 


POD  Life  Cycle  Framework  Analyses 


•Materiel  Fielding  Analysis 
•Post  Production  Support  Analysis 


AP239,  Product  Life  Cycle  Support  Activities 


•Analyze  Support  Feedback 
•Collect  Data  and  Provide  Feedback 


GEIA-STD-0007 


•All  Entities  Affected  by  Data 
Collection  and  Feedback 
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SiBJG  Picture” 


DoD  Framework 


Supportability 

Analysis 


AP239  Activities 


System  Support 


IETM 
Training 
Provisioning 
Parts  Lists 
Etc... 
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Logistics  Product  Data  Lisas 


■  Maintenance  Planning 

-  Maintenance  Plan 

-  Maintenance  Allocation  Chart 

-  Preventive  Maintenance  Checks  &  Services 

-  Maintenance  Procedures  for  lETMs  (Task  Analysis  XML  Schema) 

■  Support  and  Test  Equipment 

-  Support  Equipment  Recommendation  Data 

-  Calibration  Maintenance  Requirements  Summary 

-  TMDE  Registration 

Supply  Support 

-  Provisioning  Technical  Documentation  Lists  (Long  Lead,  Post  Conference, 
Common,  Bulk  Items,  etc.)  (Provisioning  XML  Schema  &  Style  Sheet) 

-  Design  Change  Notice  Information 

-  Cataloging/Screening/Parts  Breakout 

-  Indentured  Parts  List  (for  lETMs) 
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Logistics  Product  Data  Llsss  (Continued) 


■  Manpower,  Personnel  &  Training 

-  Qualitative  &  Quantitative  Personnel  Requirements  Information 

-  Manpower  Authorization  Criteria 

-  Task  Inventory/Training  Task  List 

-  New/Modified  Skill/Training  Requirements 

-  Identification  of  Training  Devices 

Packaging,  Handling,  Storage,  and  Transportation 

-  Packaging  and  Preservation  Data  (Packaging  XML  Schema  and  Style 
Sheet) 

-  Transportability  Requirements 

Facilities 

-  New/Modified  Facilities  Requirements 

-  Maintenance  Tasks  Requiring  New/Modified  Facilities 

■  Reliability  and  Maintainability 

-  Reliability  Centered  Maintenance  Results 

-  FMECA  Results 
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Contracting  for  Logistics  Product  Data 


Identify  the  Data  Uses  and  Analyses  Needed  for  Logistics  Product  Data 


Document  Required  Data  on  the  Attribute  Selection  Sheet 


Identify  the  Appropriate  XML  Schema  for  Data  Transfer 

-  Logistics  Product  Data 

-  Provisioning 

-  Packaging 

-  Task  Analysis 


Use  MIL-PRF-49506,  Logistics  Management  Information,  DID-ALSS- 
81529  Citing: 


-  Appropriate  GEIA-STD-0007  XML  Schema 

-  Attribute  Selection  Sheet 
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GEIA-927 


Multi-Domain  Entity  Model 

-  Lexical 

-  Graphical 

Multi-Domain  Entity 
Mapping  Tables 

Entity/Attribute  Dictionary 


GEIA-STD-0007 


Logistics  Product  Data 
Implementation  Model 

Data  Element  Dictionary 

Data  Delivery  Reqts 


ISO  10303,  AP239 


IDEFO  Activity  Model  -  PLCS 
Process  Model 


Entity/Attribute  Model  for 
PLCS 

-  Lexical 

-  Graphical 

Data  Exchange  Sets 

-  Capability  Modules 

-  Entity  Templates 

-  Reference  Data  Library 
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GEIA-STD-0007  &  DEXs 


■  AP239  DEXs  are  Developed  based  upon  Business  DEXs. 

■  GEIA-STD-0007  Represents  DoD  Business  DEX. 

■  LOGSA  Objective:  Work  with  the  Organization  for  the 
Advancement  of  Structured  Information  Standards  (OASIS)  to 
Incorporate  GEIA-STD-0007  Logistics  Data  into  the  Appropriate 
DEX’s  (1-5)  and  Create  New  DEXs  where  Gaps  exist.  (1-2  year 
Timeframe). 

Must  Retain  Ability  for  DOD  to  Contract  for  Delivery  of  Structured 
Logistics  Product  Data. 
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GEIA-STD-0007  Spiral  Development 


■  S1000D  Data  Requirements  (IETM) 

S2000M  Data  Requirements  (Provisioning) 

■  S3000L  Data  Requirements  (LSA/LSAR) 

■  Def-Stan-0060  Data  Requirements  (UK  LSA/LSAR) 

■  Documents 


-  Drawings 

-  Illustrations 

Level  of  Repair  Analysis  Data 
Improved  Task/Subtask  Referencing 
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GEIA-STD-0007  Summary 


Re-Establishes  Industry/DOD  Exchange  of  LSAR  Data 
Balloting  Completed  Sep  06  -  Resolving  Comments 

■  Publish  -  Oct/Nov  06 
Develop/Participate  in  ISO  PLCS  DEXs  -  ongoing 

■  Develop,  Ballot  &  Publish  GEIA-HB-0007  -  Dec  06 


C  onta  ct  I  nform  ail  on 


■  Jim  Colson  (USAMCLOGSA) 

-  multiview@logsa.armv.mil 

-  256-955-9928 

■  GEIA-927  &  GEIA-HB-927 

-  www.geia.org 

■  GEIA-STD-0007  (Ballot  Version) 

-  Balloted 

-  Publish  -  Oct/Nov  06 

■  GEIA-HB-0007 

-  Draft  Completed 

-  Publish  -  Dec  06 
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Backup  SiJjdea 


Backup  Slides 
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Standards  Efforts 


■  Organization  for  the  Advancement  of  Structured 
Information  Standards  (OASIS) 


-  International  Consortium  promoting  development  of 
international  e-business  standards 

-  Driving  Development  of  AP239  PLCS  Data  Exchange  Sets 
(DEX’s) 


»  DEX1 ,  Product  Breakdown  fer€upport 
»  DEX2,  Fault  States 
»  DEX3,  Task  Specific 
»  DEX5,  Maintenance 
»  DEX7,  Operational  Feedback 
»  DEX8,  Product  as  Individual 
»  DEX4/9,  Work  Package  Definition  and  Report 
»  Othen^ 


-  LOGSA  is  Voting  Member 
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ASD  S3000L 

Application  Handbook  for  the  LSA  of  Defence  Products 

AeroSpace  Defense  Association  (ASD)  Sponsored  Effort 

Companies  Committed  Resources  for  Development  of  Spec 

-  Boeing 

-  Dassault  Aviation 

-  EADS/Airbus 

-  BAE 

-  Saab  Aero 

■  Minimal  Government  participation  (Industry  Driven) 

-  UK  MoD 

-  USA  LOGSA 

Content  will  follow  a  Task  focused  approach  (FMECA,  RCM,  Task  Analysis,  etc.) 

■  Data  Requirements  will  be  defined  later  (early  07) 

-  DEX’s  (AP239  Approach) 

-  GEIA-STD-0007  data  model? 

■  One  Year  Effort  -  1  Jun  06  Start 


■  Quarterly  reviews 

-  Sep  -  SE 

-  Dec  -  FR 

-  Mar-GE 

-  Jun -US 

S4000A  (Specific  Analysis:  RCM,  LORA,  etc..)  TBD. 
Complimentary  to  S1000D  (Tech  Pubs)  and  S2000M  (Provisioning) 
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Data  Standards  and  Relationships 


Lavers  of  Normalization 


Integration  Reference  Model 
(Horizontal) 


◄ -  Application  Models 

(Vertical) 


-  Element  Definitions 

< -  Naming/Identification 


A  Reference  Model  that  integrates  across  domains,  facilitates  the  normalization  of 
information.  Each  functional  domain  must  only  validate  with  1  reference  model  and 
they  become  harmonized  with  all  other  domains. 


GEIA-927  Common  Data 
Schema  for  Complex  Systems 

James  Colson 
U.  S.  Army 

Logistics  Support  Activity  (LOGSA) 

U.S.  Army  Materiel  Command 
Logistics  Support  Activity 
Sparkman  Center ,  Bldg  5307 
Redstone  Arsenal ,  AL  35898-7466 

www.locisa.armv.mil 


1 6  Oct  2006 


Outline 


■  GEIA-927  Design/Purpose 

■  GEIA-927  Version  1.0  Content  and  Status 

■  Implementation  approach 

■  GEIA-STD-0007  Content 

■  GEIA-HB-0007  Content 


16  Oct  2006 
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GEIA-927  Purpose 


■  Facilitate  the  exchange  of  data  concepts  between 
product  data  domains  (e.g.  systems  engineering, 
product  life  cycle,  mechanical,  etc.)  for  complex 
systems  by  harmonizing  disparate  standards  into 
a  single  object  model 


■  Primary  elements 

-  Entity/Attribute  Mapping  between  domain  standards  and  the 
GEIA-927  Entity  Model 

-  GEIA-927  Entity  Model  (lexical  and  graphical) 
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GEIA-927  Version  1.0  Content 


Standard  Based  On: 

-  ISO  18876  (IIDEAS)  Basis  for  Data  Modeling  Methodology 

-  ISO  15926-2  (EPISTLE  Core  Model  (ECM))  Initial  Integration 

■  Currently  Integrates: 

-  PAS  20542  -  Systems  Engineering 

-  ISO  10303  (STEP)  AP212  -  Electrotechnical  Design  and  Installation 

-  ISO  10303  (STEP)  AP239  -  Product  Life  Cycle  Support 

-  ISO  10303  (STEP)  AP214  -  Core  Data  Model  for  Automotive  Mechanical  Design 
Processes 

-  Logistics  Support  Analysis  Record  Data 

-  Field  Logistics  Data  (Inventory,  Requisition,  Maintenance,  Usage) 

-  Logistics  Modeling  Data  (level  of  repair,  cost  analysis,  post  fielding  analysis) 


Functional  Views  of  Model 

-  Activity 

-  Common 

-  Documentation 

-  Functional  Architecture 

-  Logistics  (2) 

-  Effectivity 


-Physical  Architecture 
-Product 
-Property 
-Requirement 
-System  Architecture 
-Product  Life  Cycle 


Entity/Attribute  Model  in  HTML  Format 
Entity/Attribute  Definitions  (~800) 
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Data  Standards  and  Relationships 


Lavers  of  Normalization 


Integration  Reference  Model 
(Horizontal) 


◄ -  Application  Models 

(Vertical) 


-  Element  Definitions 

< -  Naming/Identification 


A  Reference  Model  that  integrates  across  domains,  facilitates  the  normalization  of 
information.  Each  functional  domain  must  only  validate  with  1  reference  model  and 
they  become  harmonized  with  all  other  domains. 
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Development  Process 


(Final  Version) 


(Initial  Version) 


V1.0 


GEIA-927 
VO. 5 


Log  Data 

Inventory/Requisition 


AP239 

PLCS 


GEIA-927 

V0.1 


Initial 

IM  (ISO  15926-2) 


AP212 

ElectroTechnical 


PAS20542 

Systems  Engineering 


AP214 

Automotive  Mechanical 

„  LOGSA  Models 

LORA,  Cost  Analysis,  PFSA 

>  1388-2B 

Logistics  Support  Analysis 

Log  Data 

Maintenance/U  sage/Demands 

GEIA-STD-0007 

Logistics  Product  Data 
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Example  -  927  Effectivity  View 

(Simplified) 


AP239 

PAS20542,AP212,AP239 

AP212 

AP212  &  AP239 


Effectivity 


Effectivity 

Relationship 


Time  Based 
Effectivity 


Lot 

Effectivity 


Serial 

Number 

Effectivity 


System 

Effectivity 
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Mapping  Example  -  927  System  Instance 
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Example  -  Mapping  Table  AP212 


AP212  -  Effectivity  UoF  Mapping  Table 

Form  1  -  Base  Analysis 

Form  3  -  Mapping  Information 

Application 

Element 

Number 

AM  Term 

Reference  path 

Comments 

Entity  Name 

IM  Element 

4.2.79 

date_time 

point_in_time 

pointJn_time  <= 

temporal_boundary_ 

of_state 

Existing  entity  concept. 

4.2.120 

effectivity 

effectivity 

effectivity  <=  single_individual_ 
state 

Existing  entity  concept. 

4.2.120.1 

effectivity  to  organization  (as 
concerned_ 
organization) 

effectivity_ 

organization_ 

relationship 

effectivity 

effectivity  <- 
effectivity_organization 
_relationship.valid_ 
effectivity 

effectivity_organization_relatio 

nship 

effectivity_organization_relatio 

nship 

,concerned_ 

organization  ->  organization 
organization 

This  is  an  existing  attribute 

promoted  to  entity  status 
concept  which  is 
represented  by  a  PATH. 

4.2.120.2 

description 

description_of_effectivity 

_as_ 

text.content 

/DESCRIBE  AS 

TEXT(thing)/ 

This  is  an  existing  attribute 
concept  inherited  from 
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Application  Example 


GEIA-927  Provides  a  roadmap  for  the 
Exchange  Of  Entities  between  standards/domains 


GEIA-927  Fragment 
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GEIA-HB-927,  Handbook  and  Guide  lor  GEIA-927 


■  Content 

-  Data  Modeling  Guidelines  (Principles  of  Modeling) 

-  Data  Model  Usage  Guide  (Explanation  of  EXPRESS) 

-  Integration  Procedure  (GEIA-927  Modeling  Procedures) 

-  Schema  Tailoring  Guidelines  (How  to  Apply  to  Specific 
Application) 

■  At  GEIA  for  Publication 
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fo  Sum 


arize  GEIA-927 
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Exchan  g  e  M  echanisms 


Standard  DEX  1 


•  Central  Exchange 

Federated  database,  must 
establish  exchange 
agreements  with  each 
“Partner”. 

•  Point  to  Point 

Simple  exchange,  must 
establish  exchange 
agreements  and  Protocols 
with  each  Partner 


•  Closely  Tied 

Related  exchange,  typically 
similar  data  and  structure 


GEIA-ST D-0007  Purpose 


Define  Logistics  Product  Data  Generated  During 
Design  of  a  System,  End  Item,  or  Product 

Define  Data  Exchange  Mechanisms  (A  “DEX”) 

DAU  “Wall  Chart” 


Prototypes/ 
Engineering 
Dev  Models 


Prototypes/ 
Engineering 
Dev  Models 


Proouct  Support 
Strategy 


Support  F 

■Statutory' Regulatory  'Product  Support  Elemente 
■Source  of  Support  -Supeh' Supper  I  -Joining 

■Legacy  Considerations  "  [l' 


Demonstrate  Product  S upport  Capability'1 
■Footprint  Reduction 
■Scpply  Chain  Managemenl 
■Product  Support  EJemenls  , 


DRR 


■YVtiarW  S yu  Suer  erf  t 
Hoinrenaiwe  O&^ciT'iwh  A 
.Rjflgui'dm  enw 

■COG 

•SEP 


•TEMP 


Performance  Based  Logistics  (PB-L)  Strategy  cwdu rsv/pon  Ap/rxH  \ 

Performance  Bawd  Builneic  Cace  Produc  I  Support  hlegralor.'  V" 

Agreemerifi  Analytic  ProdJd  S-dpporl  Provider 


Interpret  U 

Flefire  System 
Performance  Specs  A 
Enr=ir  on  mental  Conslnhba 


^Tradee^l 


Develop  System 
Functional  Specs  t 
System  Verification  Plan 


Logistics  &  Technical  Acronyms 


_  .  .llemdlveSyelc _ 

BLFiP-  Beyond  Lew  Rale  hllal  Produc  Ion 
CDF?-  Or  Ileal  Deslgi  Review 
01-  Conlgunllon  fern 

DTf.E  -  Developmenlal  Teel  A  Evaluation  POH  -  Preliminary  Deelgn  Bevle 

EOA-  Early  Operalonal  4s  see  amen  I  PCA  -  Phyak  J  Centigu  ration  jUj-JI 

ESOH  -  ErwIronmeriaL  Sdely  *  Occ  ipdlenal  Health  PRF;  -  Production  Frovdneei  Fsevlew 
FCA  -  Fmdlenal  Cortlguvillon  Audi  PPP  -  Program  Protection  Plan 

-  - - Etierts  A  Critically  Analyst  FCH  -  Heltsbllh . 


SalelyA  Occupational  Fteallh  Evaluation 
POH  -  Preliminary  Deelgn  Flevlew 


FNEOA  -  FalUre  Mode  E 
FOTtiE  -  Follow-on  Teel  A  Evaludlon 
FTA  -  Fell  ire  TleeAndysta 
IOTBE  -  Inlld  Coe  rational  Teel  &  Evaluation 
ISR  -  In-Servke  flevlera 
ISP-  kilermdlen  Support  Plan 
ITH-  nillalTechrlcalFtevlew 
JITC-  Jclri  hleroperarbltiy  Teel  Command 
LFT6E  -  Live  Hre  Teel  A  Bvdiullcn 
LORA-  Lewi  el  Ffcwlr Anatyele 
LR1P-  LcwFule  hlihl Production 
MTA-  Hdrienance  Tcul:  Aralyab 
0A-  Operdlonal  Aseesemert _ 


billy  Centred  Halnlenanc  e 


RMS  -  RetiaMlly.  MalrtakirtUlyA 
Supperiabllly 
SEP  -  Syeleme  En^neerhg  Plan 
SFFf-  i-yclem  Functional  Review 
SRH  -  Syden  FbquPe  merle  Ftoilew 
STA-SyelemThreal  Actennwrt 
E-'iFI  -  System  Yeriticallon  Hew  lew 
TAE  -  Ted  &  Evaludlen 
TEMP  -  Teel  *  Evaluation  Maeler  Hut 
TDS-  Technology  De-v  el  epme  it  SI  in  leg  y 
THA  -  Technology  Fteadhees  Aseeeemeit 
THR-Teei  Fieadheee  Hewlew 


Evolve  Functional 
Performance  Specs  into 
Functional  (Design  to) 
s  and  Cl  Verifioation  Plan 


Evolve  Cl  Functional 
Specs- into  Product 
(Buld  to)  Documentation 
and  Inspection  Plan 


OUTPUTS 

/  "fn  till'  Prod  SaeelViw 
■Tesi*  flewrfe 
■TEMP 

■EbjTwurta  of Pwdvct 
Suppvit 

■Risk  4i>evi ni  on  r 
■SEP  ■ ThA 
■PGSh E 
•inpufoto: 

,  -CPO  -STA  4SP 

\  ■Cwt.Vnnpcwtr  Get  / 

FCA  i- 

r  SVR  ) _ (^PRR.  > 

Combined  DTA  E'OT&E'LFTILE1 
Demons  b-rtie  System  to 
Specified  User  Needs  & 
Environ  mental  Constraints 


System  DUE.  LFTiEtOAft. 

Voril  r  Sye  bm  Func  lionni  ty 
A  Constraints  Complianco 
|  Speoe 


Integrated  DTAE.  LFTS.E  ft 
EGAs  Verily  Performance 
Compliance  to  Space 


Indh  iduai  Cl 
Verification 
DT-ftE 


Fabricate.  Assemble, 
Code  to  "Buid-to" 
Documentation 
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GEIA-STD-0007,  LdcjjsIjds  Product  Delhi 


■  Content: 

-  Implementation  Model  of  the  Logistic  Support 
Analysis  Record  (LSAR)  Data 

-  LSAR  XML  Schema 

-  LSAR  Data  Definitions 

■  Objectives: 

-  Provide  Industry  Standard  for  Acquisition  and 
Exchange  of  Logistics  Product  Data  by 
Industry/DoD 

■  Publication  Timeframe: 

-  Initial  Balloting  Completed  19  Sep  06 

-  Re-Ballot  in  Nov  06  per  Request  of  Air  Force 
and  Navy 


16  Oct  2006 


15 


GEIA-ST  D-00  07  Content 


Functional  Area  Entities/Attributes 

-  Cross  Functional  Requirements 

-  Operations  and  Maintenance 

-  Reliability  Requirements  and  Analysis 

-  Task  Analysis 

-  Skill  and  Training 

-  Support  Equipment 

-  Unit  Under  Test 

-  Facility 

-  Transportability 

-  Provisioning  and  Cataloging  Requirements 

■  Data  Model-  Graphical  View 

■  Data  Element  Dictionary 

■  XML  Schema  for  Logistics  Product  Data 

■  XML  Schemas  for  Transaction  Sets 

-  Provisioning  &  Style  Sheet 

-  Packaging  &  Style  Sheet 

-  Task  Analysis 

LCN,  ALC,  UOC  Assignment  Guidance 


16  Oct  2006 
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GEIA-HB-0007 


■  Overview  of  how  (e.g.  what  analysis)  and  when  Logistics 
Product  Data  is  generated  during  the  development  process 
(AP239  and  DOD  Lifecycle  Models) 

■  Description  of  the  Logistics  Product  Data  Entities  and 
Attributes  (When  Required,  Sources,  Indenture  Level 
Relationships,  Primary  Use) 

■  Contracting  for  Logistics  Product  Data 

■  Appendices 

-  Sample  Relational  Tables 

-  Test  Data  Set 

-  LCN,  ALC  and  UOC  Guidance  to  Include  Relationship  to  S1000D  SNS 

-  Data  Cross  Reference  List  (0007,  DEF  STAN  00  60,  MIL-STD-1388-2B) 

Publish  Dec  06 


16  Oct  2006 
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GEIA-927 


GEIA-STD-0007 


Logistics  Product  Data 
Implementation  Model 

Data  Element  Dictionary 

Data  Delivery  Reqts 


ISO  10303,  AP239 


■  Multi-Domain  Entity 

■  IDEFO  Activity  Model  - 

Model 

PLCS  Process  Model 

-  Lexical 

-  Graphical 

Entity/Attribute  Model  for 
PLCS 

■  Multi-Domain  Entity 

Mapping  Tables 

-  Lexical 

■  Entity/Attribute 

-  Graphical 

Dictionary 

Data  Exchange  Sets 

-  Capability  Modules 

-  Entity  Templates 

-  Reference  Data  Library 


16  Oct  2006 
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Summary 


■  GEIA-927 

-  Facilitates  Exchange  of  Data  Concepts  between  Product 
Data  Domains 

-  Publication  Pending 

■  GEIA-HB-927 

-  How  Standard  Was  Developed 

-  How  to  Implement  Standard 

■  GEIA-STD-0007 

-  Re-Establishes  Industry/DOD  Exchange  of  LSAR  Data 

■  GEIA-HB-0007 

-  Establishes  Relationship  of  Analysis  to  the  Data 

-  Establishes  Relationship  of  Data  to  Logistics  Products 

-  How  to  Contract  for  GEIA-STD-0007 

■  Working  Toward  AP239  Business  DEX 


16  Oct  2006 
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C  onta  ct  I  nform  ail  on 


■  Jim  Colson  (US  AMC  LOGSA) 

-  multiview@loqsa.armv.mil 

-  256-955-9928 

■  GEIA-927  &  GEIA-HB-927 

-  www.qeia.org 

■  GEIA-STD-0007  (Draft  Version) 

-  Re-Ballot  Version  in  Nov  06 

■  GEIA-HB-0007  (Draft  Version) 

-  Ballot  Version  in  Nov  06  (Concurrent  with  Standard) 


16  Oct  2006 
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Backup  Slides 


Backup  Slides 
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International  Standards  Efforts 

■  Organization  for  the  Advancement  of 
Structured  Information  Standards  (OASIS) 

-  International  Consortium  promoting  development 
of  international  e-business  standards 

-  Driving  Development  of  AP239  PLCS  Data 
Exchange  Sets  (DEX’s) 


»  DEX5,  Maintera^de  Plan 
»  DEX7,  Opora'ional  Feedback 
»  DEX8,  @iduct  as  Individual 
»  DEX4/9,  Work  Package  Definition  and  Report 
»  Others 


-  LOGSA  is  Voting  Member 


16  Oct  2006 
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ASD  S3000L 

Application  Handbook  for  the  LSA  of  Defence  Products 

AeroSpace  Defense  Association  (ASD)  Sponsored  Effort 

Companies  Committed  Resources  for  Development  of  Spec 

-  Boeing 

-  Dassault  Aviation 

-  EADS/Airbus 

-  BAE 

-  Saab  Aero 

■  Minimal  Government  participation  (Industry  Driven) 

-  UK  MoD 

-  USA  LOGSA 

Content  will  follow  a  Task  focused  approach  (FMECA,  RCM,  Task  Analysis,  etc.) 

■  Data  Requirements  will  be  defined  later  (early  07) 

-  DEX’s  (AP239  Approach) 

-  GEIA-STD-0007  data  model? 

■  One  Year  Effort  -  1  Jun  06  Start 


■  Quarterly  reviews 

-  Sep  -  SE 

-  Dec  -  FR 

-  Mar-GE 

-  Jun -US 

S4000A  (Specific  Analysis:  RCM,  LORA,  etc..)  TBD. 
Complimentary  to  S1000D  (Tech  Pubs)  and  S2000M  (Provisioning) 


16  Oct  2006 
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Example  -  japping  Table  PAS20542 


AM  element 

Number 

AM  element 

IM  element 

Reference  path 

Comments 

49 

CONFIGURATION_ELEMENT_ 

VERSION 

configuration_ 
element  version 

configurationelementversion  <= 
classofclass 

New  entity. 

Attributes  of  this  entity  are  not  mapped  at 
this  time.  The 

configuration_element_version  is 
an  PAS20542  concept  whose 
integration  unit  has  not  yet  been 
integrated  into  Multi  View. 

The  above  issue  referred  to  in  this 

comments  column  will  remain  an 
open  issue  as  of  the  revised 
document  date  noted  above  as  that 
specific  Uof  referred  to  was  not 
touched  upon  as  of  the  end  of 

Multi- view  Year  Two  efforts. 

59 

DATATRANSFER 

datatransfer 

datatransfer  <=  relation 

New  entity. 

PAS20542  note  on  data  transfer  indicates 
that  it  may  be  used  to  depict  a 
physical  interaction  as  well  as  data 
interactions. 

With  regards  to  the  above  comments,  after 
the  analysis  of  the  Physical  and 
Functional  architecture  Uofs,  it 
turns  out  that  the  composition  of 
function  instances  pointed  to  by 

16  Oct  2006 
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GEIA-927  Relationships 


Reference  Model 


Implementation  Model  Implementation  Model 


Etc. 


16  Oct  2006 


25 


AP239 

PAS20542 


Requin 

Defin 

^ment 

ition 

Requirement 

Occurrence 

(Simplified) 


Thing 


System 

View 


System 

Definition 


Integrates  concepts  from 
Systems  Engr.  and  PLCS 


Partial 

System 

View 

E 

Requirement 

Instance 

16  Oct  2006 
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0EIA-927  Integration  Process 


•  Entities  from  a  standard  or  product  area  are  organized  into  a 
series  of  relationship  diagrams. 

•  The  diagrams  are  parsed  and  converted  into  a  format  suitable 
for  analysis. 

•  Each  entity  and  its  related  attributes  are  abstracted  from  an 
application  specific  level  to  a  general  level  to  fit  into  the  GEIA- 
927  model. 

•  Entities  are  analyzed  considering  existing  concepts,  superseded 
concepts  or  new  concepts  to  test  fit  into  the  GEIA-927  model. 

•  Upon  completion  of  analysis,  entities,  attributes  and  entity  to 
entity  relationships  are  added  to  the  GEIA-927  master  model. 


16  Oct  2006 
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Views 


Views  allow  users 
to  “digest”  model 


Requirements 


Activity 

Effectivity 

Products 

System  Arch 

1 

mnctional  Arcl 

1 

Physical  Arch 

Documentation 

Pro] 

perty — 

sties 

Express-G 

j^OgE 
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GEIA-927  Views — 


GEIA-927  Integration  Results 

PAS24502_ AP239_ AP212_ Log  Data 


Activity  View 

Common  View  Common  View  Common  View 

't' 

Documentation  View 


Activity  View 
Common  View 


Documentation  View 


Effectivity  View 


Effectivity  View 


Effectivity  View 


Functional  Architecture  View 


Functional  Architecture  View  Functional  Architecture  View 


Logistics  View 


Physical  Architecture  View 


Physical  Architecture  View 


Product  View 
Property  View 
Requirement  View 


Product  Life  Cycle  View 
Product  View 
Property  View 
Requirement  View 


Product  View 
Property  View 
Requirement  View 


Product  Life  Cycle  View 
Product  View 
Property  View 
Requirement  View 


System  Architecture  View  System  Architecture  View 
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GEIA  927 


AP239 

PAS20542 

AP212 


Mapping  of  Entities  Provides  Roadmap 
for  Exchange  of  Data  Between  Standards 


16  Oct  2006 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


System  Engineering  and  Integration 
for  Submarine  Combat  Systems  in 
the  COTS  Environment 


Dennis  J.  Cooper 
SWFTS  Program  Manager 
Lockheed  Martin  Integrated  Systems 

Joan  Sienkiewicz 
SWFTS  Program  Manager 
General  Dynamics  Electric  Boat 

Mike  Oliver 
SWFTS  Program  Manager 
Lockheed  Martin  Maritime  Systems  and  Sensors 
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Submarine  Combat  System  Paradigm  Shift 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


•  Submarine  combat  systems  employ  Commercial-Off-The- 
Shelf  technology  and  open  system  standards  to  provide: 

-  An  affordable  path  to  leverage  commercial  technology  in  parallel  with 
the  build-test-build  development  cycle 

-  Asynchronous  rapid  transition  of  advanced  functional  improvements 
through  modular  software  builds  in  a  networked  environment 

•  Harnessing  this  capability  required  a  change  to  the  system 
engineering,  interface,  integration,  and  configuration 
management  processes 


This  SE&I  Model  Was  Proven  Successful  on  the 
Virginia  Class  Combat  System 


Today’s  Dynamic  Environment 


LOCKHEED  MAR 
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GENERAL  DYNAMICS 

Electric  Boat 


•  Varying  acquisition  “cycle  times”  between  subsystems 

•  Fleet  driven  rapid  introduction  of  warfighting  capability 

•  Technology  transition  from  custom  to  COTS 

•  Radical  reductions  and  gaps  in  funding 

•  Spiral  Development 

•  Increasing  system  complexity 

•  Software  Intensive  Systems 

•  Desire  for  commonality  across  platforms 

•  Network  Centric  /  Interoperability 

•  Information  Security  and  Assurance 


Business  Models  have  evolved  to  adapt  to  the 

changing  environment 


Architecture  Contrast 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


Closed  System 


Open  System 


•  Use  of  closely  held,  private 
interfaces,  languages,  data  formats 
and  protocols 

•  Critical  importance  given  to  unique 
design  ana  implementation 

•  Use  of  individual  company 
preferences  to  set  ana  maintain 
interface  specifications 

•  Vendor  and  technology  dependence 

•  Difficult  and  more  costly  integration 

•  Use  of  sole-source  vendor 

•  Expansion  and  upgrading  usually 
requires  considerable  time,  money 
and  effort 

•  Components,  interfaces,  standards, 
and  implementations  are  selected 
sequentially 


•  Use  of  publicly  available  and  widely 
used  interfaces,  languages,  data 
formats  and  protocols 

•  Critical  importance  is  given  to 
interface  management  and  widely 
used  conventions 

•  Use  of  group  consensus  process  to 
maintain  interface  specifications 

•  Vendor  and  technology 
independence 

•  Easier  and  more  cost  effective  to 
integrate 

•  Use  of  multiple  vendors 

•  Easier,  quicker  and  less  expensive 
to  upgrade 

•  Components,  interfaces,  standards, 
and  implementations  are  selected 
interactively 


Interface  Management  is  the  Key  to  Success 


Source:  ITEA  Journal:  September/October  2001  (Volume  22,  Number  3) 
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Open  System  Model 


LOCKHEED  MAR 
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GENERAL  DYNAMICS 

Electric  Boat 


•  Focus  on  the  network  and 
subsystem  to  subsystem 
domains  to  define  and  manage: 

-  Open  Architecture  /  Open  System 
Standards 

-  Interface  Design  and  Definition 

•  Functional  not  physical 

-  Network  Design,  Services  and 
system  performance 

-  Network  software  solutions 

-  Technology  Evolution 


System  Engineering  &  Integration 

Common  Network 

System  Engineering  /  Performance  Requirements  /  Interface  design  j 

Integration  /  Validation 

i  ♦  ,  +  +  +  ,  i 


Legacy 

Legacy 

Legacy 

Legacy 

New 

New 

New 

New 

-  Standardized  processes  and 
products 


This  Open  System  Model  has  been  implemented  on  the  Submarine 
Functional  Interface  Baselines  by  a  disciplined  and  measured 
approach  to  Interface  development  and  data  flow  management 


Virginia  Class  Submarine  Combat  System 


LOCKHEED  MAR 
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GENERAL  DYNAMICS 

Electric  Boat 


ULAN 


O 


Need  to  manage  and  control  the  interfaces  for  the  23  subsystems  of 
the  Virginia  Class  Submarine  Combat  System 
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It  Starts  with  Requirements 
Management 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


SE&I  provides  Top  Level  Combat  System  and  Interface  requirements. 
All  other  requirements  are  managed  by  the  Federated  subsystems 


r 

REQUIREMENTS 

< _ 

Fleet  Requested 
Improvements 

APB  Requirements 

FORCENet 

Requirements 

Sea-Test  Results 

Technology 

Improvements 

Open  Architecture 
Standards 

PARM  Input 

IA  Requirements 


SYSTEMS  ENGINEERING  IPT 


ANALYZE  NEW 
REQUIREMENTS 


V 


FUNCTIONAL 
DECOMPOSITION  & 
ALLOCATION  OF 
PLATFORM  /  INTER' 
SUBSYSTEM  LEVEL 
REQUIREMENTS 
AS  AN  IPT 


Internet 


Web  Interface  Product 
Tool  (WIPT) 

IDL  GRL 

Trade  Study 
^commendations 


Allocated 
Requirements 


Baseline  Changes 
via  Change  Board 
Process 


EXECUTION  AGENTS 


PROGRAM  MGMT  IPT 


SYSTEMS  ENGINEERING 
IPT 


HW/SW  TECHNOLOGY 
INSERTION  IPT 


INTEGRATION  TEST  & 
EVALUATION  IPT 


SUPPORTABILITY  IPT 


FEDERATED  SUBSYSTEM 
MANUFACTURERS/PARMS 


A  Streamlined  System  Engineering  Approach 
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Evolution  of  Interface  Standardization 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


SUBSYSTEM  A 
'  SUBSYSTEM B 

SUBSYSTEM C 
-INVOLVED  IN  IDL  WG 
-  USE  COMMON  AGREED 
J  -  TO  IRS  &  IDD  FOR 
DESIGN 

-USE  COMMON  IDL 
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System  Engineering 

Network  Architecture,  Interface  Design 


LOCKHEED  MAR 
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GENERAL  DYNAMICS 

Electric  Boat 


Today 


Network  /  Group  Product  Design  Process 

Subsystem  A  Subsystem  B 


Subsystem  C 


Subsystem  D 


• Sonar  Settings 
•Streams 


Group  Products 

•  Open  System  Architecture 
Requirements  and  Standards 
(OSA) 

•  Group  Requirements  List 
(GRL) 

•  Interface  Definition  Language 
(IDL) 

•  Group  Data  Dictionary  (GDD) 

•  System  Network  Design 


Interface  Working  Groups 
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SE&I  Interface  Product  Definitions 


LOCKHEED  MAR 
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GENERAL  DYNAMICS 

Electric  Boat 


•  SE&I  Interface  Products  under  Configuration  Management: 

-  Group  Requirements  List  (GRL) 

•  Database  that  contains  all  interface  requirements  for  all  subsystems. 

-  Group  Data  Dictionary  (GDD) 

•  Contains  definitions  for  all  Interface  Data  Elements  (IDEs)  and  IDE  Assemblies 
(IDEAs)  produced  and  consumed  by  subsystems  within  a  Data  Group 

-  Interface  Definition  Language  (IDL) 

•  IDL  provides  the  standard  interface  between  objects,  and  is  the  base  mechanism 
for  object  interaction  between  subsystems. 

-  Interface  Integration  Data  Base  (IIDB) 

•  Provides  details  of  what  interfaces  are  required  for  each  subsystem  on  a  per  hull 
basis.  Grouped  by  interface  types  and  contains  the  Interface  /  method  /  Name 
Space  Name  (CORBA),  Service  Name  (Services),  Subsystems,  GRL  linkages, 
GDD  linkages  and  indications  of  Provider,  Recipient,  Originator. 
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SE&I  Product  Relationships 


LOCKHEED  MAR 
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GENERAL  DYNAMICS 

Electric  Boat 


GRL 


Request  For  Subsystem 
Requirement 
Change/Enhancement 

I 

Facilitators  develop  requirements 
and  products  based  on  request 
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Sequence  No. 


Subsystem 

Allocation 


Facilitators  work  with  subsystems 
to  generate  IDL  per  requirement 

IDL 


IDL  Embedded 
In  Subsystem 
Application 


t*~  @lile  30urKling:Dep1hs_cs  nJi 

■  Classification-  UNCLASSIFIED 

*  ©version  501  August  2001  initial 

"j1 

flifndsr  ■sound  mg  Depth  s_cs._id.1_ 

^define  _s*undinaDeptlis_'CS_ldl_ 

^include-  "rtavtgatton_t_icl_ 
i" 

■  DescfSJtiwn  The  fiolidwing  is  the  ESdinmon  for  Che  sounding  depDra 

*  clientfeerver  interface. 

V 

interface  &oundmg  D@p"ihg_cs  { 


'  Description 

'  ©return  na«9al»ofli_t :  sou  ridings  De-pHTS-j 


navigarion_c  sound  ng50epths_tgetSoundlngOep€h^{  ); 
];  (i  end  sounding CNepths_CS  ' 

#endif  Jt  _soun dug De ptfr5_C5_ idl_ 


GDD 


J^Auto  generated  from  IDL 


IDL  File  Name 

Method(s)  Producing  Data 

npes Senso  rRep  ortE  vents ev.  i  dl 

npes_SensorReportEvents_ev::esmSensorReportEvent_ev:: 
newEsmSenso  rRep  ortE  vent . 

npes_Senso  rRep  ortEvents_e  v: :  ima  gingS  enqgtSepo  rtE  vent_ 
e  v:  :newlm  agin  gSensorReportE  vent,  * 

npes_Senso rRep  ortEvents_e v::radarSenso rRep ortEvenl_ev. 

:  newRad  arSe  nsorRep  ortE  vent . 

npes_Senso  rRep  ortEvents_e  v::  son  arSensorRep  ortE  vent_ev: 
:newSonarSensorReportEvent, 

npes_Senso  rRep  ortEvents_e  v: :  srwsSensorRepo  rtE  vent_e  v: : 
newS  rwsSenso  rRep  ortEvent 

npes_Senso  rRep  ortE  vents_ev.  i  dl 

npes_Senso  rRep  ortEvents_e  v::esmSe  nsorRep  ortEvent_ev: : 
newEsmSenso  rRep  ortEvent, 

npes  Senso  rRep  ortEvents_e  v::  ima  gingS  ensorRepo  rtE  vent_ 
ev::newlmagingSensorReportEve  nt, 

npes_Senso  rRep  ortEvents_e  v::radarSenso  rRep  ortEvent_ev: 

:  newRad  arSe  nsorRep  ortE  vent , 

npes  Senso  rRep  ortEvents_ev: :  son  arSe  nsorRep  ortE  vent_ev: 
:newSonarSensorReportEvent, 

npes_Senso  rRep  ortEvents_e  v: :  srwsSensorRe  po  rtE  vent_ev: : 
newS  rwsSenso  rRep  ortE  vent 

Method: 
programmed 
procedure  within 
an  interface 


Linked  to  a 
particular 
interface  data 
rate 


V 

Subsystem 

Allocations 
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Interface  Working  Group  Participants 


GENERAL  DYNAMICS 

Electric  Boat 


Infrastructure: 

Navigation: 

Contact  Data: 

On-Board  Training: 

Hull/Mechanical  Data 

Communications: 

Environmental  Data: 

Lockheed  Martin 

Lockheed  Martin 

Raytheon 

Electric  Boat 

Electric  Boat 

Electric  Boat 

Lockheed  Martin 

Progeny 

Electric  Boat 

GD-AIS 

NUWC 

Northrop  Grumman 

GD-AIS 

Lockheed  Martin 
Kollmorgan 

NUWC 

Lockheed  Martin 

NUWC 

Lockheed  Martin 
NUWC 

SPAWAR 

GD-AIS 

Electric  Boat 

LOCKHEED  MAR 


•  Integrated  Product  Team  structure  to  develop  and  maintain  interface  specifications 
and  products  (GRL,  IDL,  GDD,  Network  Design,  Information  Assurance) 

•Adopts  widely  used  standards,  data  formats,  and  protocols 

•  Defines  key  attributes  of  the  network  and  services 

•Consensus-based  interface  definition  and  design _ 


“The  key  is  in  designing  an  architecture  that  is  going  to  take  advantage  of 
commercial  standards ,  the  ability  to  pull  pieces  out  and  reuse  them  in  other 
systems  and  platforms ,  and  that  allows  third  parties  access” 
Delores  Etter,  Assistant  Secretary  of  the  Navy  for  RDA 
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Interface  Layers 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


Yesterday  ■■="  ;  -  -  Today 


SE&I  Toolset 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


IS 


VL 


Requirements 


Web  Interface  Product 
Tool  (WIPT) 

IDL 


Tactical  HW  Interface 
Platform  D/B  (TIPD) 

ICD 


a 


Develop 


Web  Integration  and 
Test  Tool  (WITT) 


91 


Test 


Debug  PTRs 


Test 


Supportability 
Integrated  Logistics 
Capability  (SILC) 


Support 


Fully  integrated  and  web-enabled  toolset  for  SE&I 
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What’s  next? 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


•  It  doesn’t  end  with  Interface  Management! 

-  Systems  engineering  processes  and  products  must  be 
balanced  with  integration  products  and  processes 

-  There  is  an  inherent  relationship  between  standards, 
interface  engineering  and  system  integration 

•  System  definition 

•  Interface  design  and  development 

•  System  test  and  integration 

-  The  confluence  of  an  open  systems  architecture,  standards 
and  system  integration  leads  to: 

•  Easier  technology  insertion 

•  Fielding  capability  faster 

•  Streamlined  system  integration 


SE&I  is  a  balance d  approach  between  interface 
management,  standards  and  system  integration 


15 


SE&I  Process  and  Products 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


Products: 

SRVM 

Test 

Procedures 
Test  Results 
PTRs 


Interface 

Development 


Fully  integrated  and  web-enabled  toolset  for  SE&I 
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Test  and  Integration 
(The  “I”  in  SE&I) 


Goals: 

•  Achieve  a  single  integrated,  cohesive 
working  Combat  System 

•  Risk  Mitigation 

•  Cost  containment 

•  Schedule  Control 

•  Optimize  Government  and  Industry 
Lab  resources 

•  Early  integration  and  test  of  interfaces 

•  Pull  integration  and  Lifetime  Support 
issues  from  the  fleet  and  solve  in  labs 


GENERAL  DYNAMICS 

Electric  Boat 


Approach: 

•  Virtual  Integration  Facilities  to 
optimize  capital  investments 

•  Structured  interface  testing 

•  Allow  collaborative  debug  of  problems 
from  development  lab 

•  Distribution  of  software  builds/patches 

•  Remote  access  to  development 
environments 

•  Responsiveness  to  Fleet  needs 

•  Support  Certification 


Networked  Labs  and  Collaboration  Provide  Mechanisms  for  Early 

Integration  Testing  to  Minimize  Risk 
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Automated  SE&I  Toolkit 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


SE 


Web  Interface  Product  Toolkit  (WIPT) 


Data  Dictionary  (GDD) 


Behavior :  Method  Call: 


Platforms 


Subsystems 


Requirements 


I 


Test  Cases 


Facilities  Plan 


PARM 

Drop 

Plan 


Test  Schedule 


IDE  Web  Portals 


Requirements 


Problem  Database 


Integration  Web  |ntegration  and  Test  Too|kit  (WITT) 


Web-based  Integration  and  Test  Toolkit  (WITT)  Benefits 

■  Based  on  OSI  Model:  Expandable  Environment 

■  Automated  Test  Procedure,  SRVM,  and  ROA:  Built-in  Test  Reuse 

■  Early  System  T&l  Planning  Suite  with  Auto-Lab-Utilization 

■  Universal  Interface  Debug  Tools 

■  Online  Test  Pass/Fail  Recording  /  Automated  V&V 

■  Dashboard  Style,  Drill-Down  Technical  and  Programmatic  Metrics 


IPT  Hierarchy  &  Information 

Relationship 


SRVM / VDD 

Test  Procedures 

13 

Schedule 

I— K 

CD 

Facilities  Plan 

D 

CD 

r+ 

Test  Results 

PTR  Info 

Debug  Info 

► 

Dashboar 

Metrics 


Technical  Issues 
&  Direction 


Working 
jctmical  Data 


LTS  IPT 


Web-Enabled  SE&I 

/ 


Toolkit  Connects  all  Stakeholders 

\ 
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System  Engineering  &  Integration  Approach: 
Test  What  has  Changed;  When  it  is  ready 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


INTERFACE 
Change  Request 


Requirement  Change 


Implementation  Problem 


liikNiina 

Ihu,  J 

JmW  |Nb  D|i 

1  Ifth 

iMv'P ,  J 

■ 

mm 

ptuliit 

I'Mi' 

J 

; 

Subsystem 

Schedules 


•Prioritization 

•Debug 

•Problem  Isolation 

•Resolution 

•Verification 


Interface 

Requirements 


Group 

Products 


Test  Case 
Matrix 


Subsystem  Assessment 


~  =  >  o  CQ  Q  s 

03  O  <  <  =>  <  to 

I—  CO  Z  Q£  C/3  Q£  LLI 


Test 

Schedule  & 

Conduct 

PTR 

Facilities  Plan 

Adjudication 

xxx-750  Update  to  IPv06 

1 1 32-  Net-Centric  Warfare  (NCW)  Distance  Support 

1 1 33-  Digital  Navigation  Computer  to  Computer  I nformation  T ransfer 

xxxx-  Consolidation  of  Displays 

968-  GRL  Mapping  in  IIDB  is  incorrect 

047-  Add  T ransverse  coordinate  system  data  elements  to  sensor  and 

750  contact  reports 

Other  Network  Loading 

Other  Off-Hull  Bandwidth 

Other  Taclan  Switch  Changes 


Wd»  Aim  aii44<ai*p«i  J=«i*ry 


B  H  m 

Facility  #1  ~  Ll 


WAIF 


Facility  #2 


System  Integration  approach  has  evolved  through  several  iterations  and  provides 
complete  management  of  Requirements.  Tests.  Facilities .  Drops .  and.  PTRs 


"W 


Lessons  Applied 


LOCKHEED  MAR 


TTir^' 


GENERAL  DYNAMICS 

Electric  Boat 


•  Be  prepared  for  a  major  cultural  change  and  manage  it! 

•  Adopt  standards  widely  developed/recognized  by  industry 

•  Focus  on  interface  management  not  subsystem  development 

•  Strive  for  consensus-based  interface  definition  on  all  key  interfaces 
early 

•  New  business  models  and  processes  will  be  required 

•  Link  functionality  updates  (APBs)  to  bundled  interface  baseline  updates 

•  Be  prepared  to  migrate  with  technology  (standards  lag  the  technology 
curve) 

•  All  interfaces  are  not  created  equal 

•  Consider  and  develop  a  strategy  for  information  assurance,  fault 
management  and  fail-over  early.  Most  standards  ignore  these  areas! 

•  Plan  for  change! 


Interface  management  is  the  key! 
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1 1 II 


Lessoned  Learned  with  System  Integration  -  1 


Lessons  Learned  with 
Integration  of  Federated 
Systems  on  Legacy 
Aircraft 


Adam  McCorkle 
GTRI/ELSYS 

adam.mccorkle@gtri.gatech.edu 


Kendal  Hinton 
GTRI/ELSYS 

kendal.hinton@gtri.gatech.edu 


Georgia 
lech 


Phone:  404-407-6427 


Phone:  404-407-6042 


F-16C+  Fighting  Falcon 


•  F-16C+ 

•  F-1 6  Preblock  40 

•  Blocks  25,  30,  and  32 

•  Approximately  600  Active  Aircraft 

•  ANG 

•  AFRC 

•  ACC 

•  EW  Suite 

•  ALQ-21 3  Countermeasures  Set  (CMS) 

•  ALR-69  Radar  Warning  Receiver  (RWR) 

•  Electronic  Countermeasures  (ECM)  Pods 

•  ALQ-131 

•  ALQ-184 

•  ALE-47  Countermeasures  Dispenser  System  (CMDS) 

•  Partial 

•  Theater  Airborne  Reconnaissance  System  (TARS) 

•  ALE-50  Towed  Decoy 


Georgia  j I  ^te@®$[r©[]T] 
lech  I  OrasaffiMt© 


Lessons  Learned  with  System  Integration  -  2 


A-10  Thunderbolt 


•  A-10A 

•  “Warthog” 

•  A-10C 

•  Upgrade  to  Avionics,  Weapons,  and  Cockpit  Display  Systems 

•  Approximately  400  Active  aircraft 

•  ACC 

•  AFRC 

•  ANG 

•  EW  Suite 

•  ALQ-213  CMS 

•  ALR-69  RWR 

•  ECMPods 

•  ALQ-131 

•  ALQ-184 

•  AAR-47  Missile  Warning  System  (MWS) 

•  ALE-47  CMDS 

•  Comet  IR  Dispense  System 


Georgia 

Tech 


■ 


Lessons  Learned  with  System  Integration  -  3 


ALQ-213  Countermeasures  Set  (CMS) 


•  Replaces  All  Existing 
Discrete  EW  Control  Panels 

•  Single  Point  of  Control  of  All 
EW  Systems 

•  Semi-Automatic  and 
Automatic  Capabilities  That 
Adapt  to  Threat  Environment 


Georgia 
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II  I  1 1 II 


ALR-69  Radar  Warning  Receiver 

(RWR) 

•  Purpose  -  Detect,  Identify, 
and  Output  Visual/Audio 
Information  for  Friendly 
and  Hostile  RF  Radar 
Emitters 

•  Quadrant  Detection 

•  Visual  Output  of 
Symbology  and  Angle  of 
Arrival 

•  Audio  Output  via  Aircraft 
Intercom 


Georgia  j  ,  I 
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Lessons  Learned  with  System  Integration  -  5 


Electronic  Countermeasures  (ECM) 

Pods/Jammers 


•  Purpose  -  Provide 
Defensive  Protection  for 
Aircraft  Through  Defeat 
or  Deception  of  Radar 
Systems  Attempting  to 
Track  the  Weapons 
Platform 

•  ALQ-131 

•  ALQ-184 


ALQ-131  ECM  Pod 


Georgia , 
Tech 


,  [EtegcssmsIK) 
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Lessons  Learned  with  System  Integration  -  6 


I  I  I  I  II 


ALE-47  Countermeasure  Dispense 

System 


•  Purpose  -  Provide 
Defensive  Protection  for 
the  Weapons  Platform 
Through  Ejection  of 
Materials  Designed  to 
Distract  or  Decoy  RF  and 
IR  Based  Systems. 


•  Chaff  -  Counters  RF- 
Based  Tracking 
Systems 


Georgia 

Tech 


•  Flares  -  Counters  IR- 
Based  Tracking 
Systems 


Lessons  Learned  with  System  Integration  -  7 


AAR-47  Missile  Warning  System  (MWS) 


•  Purpose  -  Detect  and 
Identify  IR  Emissions  Near 
the  Weapons  Platform, 
and  Output  Visual 
Information  to  the 
Aircrew. 

•  Automatically  Triggers  an 
Appropriate 
Countermeasure  to 
Protect  the  Pilot  and 
Aircraft 


Georgia 


Lessons  Learned  with  System  Integration  -  8 


Theatre  Airborne  Reconnaissance 

System  (TARS) 


•  Purpose  -  Provide  Under- 
the-Weather,  Medium  to 
High  Threat,  Daytime 
Imagery  Collection 

•  Tactical  Reconnaissance 
Pod  System 


Georgia  j I 
lech  || 


Lessons  Learned  with  System  Integration  -  9 


ALE-50  T owed  Decoy 


■■■ 


•  Purpose  -  Provide  a 
Radar  Target  Decoy  for  an 
Incoming  Missile 

*  Towed  Behind  the  Aircraft 
When  Deployed  and 
Protects  the  Aircraft 
Against  RF-Homing 
Missiles  by  Drawing  Them 
Away  From  the  Host 
Aircraft 


Georgia 


Lessons  Learned  with  System  Integration  - 10 


COMET  Dispenser  Pod 


•  Purpose  -  Provides 
Extended  Duration 
Protection  for  Aircraft  in 
Infrared  Threat 
Environment 

•  MIL-STD-1553  Control 
Interface 

•  Developmental  System 
Not  Currently  Fielded 


Georgia 
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History  of  System  Integration 


•  1950s  and  1960s  -  Aviation  Electronics  are  Mainly 
Analog  and  Stand-alone  Systems 

•  Late  1960s  and  1970s  -  Became  Necessary  to  Share 
Information  Between  Systems 

*  Accomplished  Primarily  via  Discrete  Analog  Signal 
Interfaces  and/or  A/D  Converters 

*  Required  Increasingly  Complex  Wiring  and 
Connections 


Georgia 


Lessons  Learned  with  System  Integration  - 12 


History  of  System  Integration 


•  Late  1970s  -  Advent  of  Digital  Technology  Made 
Possible  Increasingly  Sophisticated  Systems 
Powered  by  Computers 

*  Led  to  a  Need  for  Higher  Speed  Transfer  of  Data 
Between  Systems 

*  Digital  Interfaces  Conceived  to  Address  this  Problem 
But  With  an  Increase  in  the  Amount  of  Wiring  and 
Connector  Complexity  Required 

*  Eventually  the  Need  Arose  for  a  Transmission  Medium 
that  Would  Allow  Multiple  Systems  to  Share  a  Single, 
Common  Set  of  Wires 


Georgia 


Lessons  Learned  with  System  Integration  - 13 


Purpose  of  MIL-STD-1553 


•  MIL-STD-1553  Was  Developed  to  Provide  a  Standard 
Solution  that  Would  Allow  Multiple  Systems  to 
Share  a  Single  Common  Set  of  Wires  (Data  Bus)  in 
Order  to  Efficiently  Move  Data  Between  Systems 


Georgia , 
Tech 
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Lessons  Learned  with  System  Integration  - 14 


Purpose  of  MIL-STD-1553 

Discrete  Analog  Signals 


Purpose  of  MIL-STD-1553 


Georgia  ^sscssmsIK) 


Lessons  Learned  with  System  Integration  - 16 


History  of  MIL-STD-1553 


*  Late  1960’s  -  H-009  Data  Bus  Specification  Used  on 
the  F-15  (Early  Forerunner  of  1553) 

*  1968  -  SAE  Establishes  Subcommittee  to  Define  a 
New  Serial  Bus  Standard  for  the  Military  Avionics 
Community 

•  1970  -  First  Draft  (From  SAE/A2K)  Developed 

•  1973  -  MIL-STD-1553  Released 

•  USAF  Only  Version  of  the  Standard 

•  Primary  User  was  the  F-16 


Georgia 


Lessons  Learned  with  System  Integration  - 17 


History  of  MIL-STD-1553 

•  1975  -MIL-STD-1553A  Released 

•  First  Tri-Service  Version  of  the  Standard 

•  F-16 

•  AH-64A 

•  1978  -  MIL-STD-1553B  Released 

•  Standard  “Frozen”  at  Revision  “B” 

•  Allowed  for  Component  Manufacturers  to  Develop  Standard 
Products  That  Provide  for  Easy  Implementation 

•  1996  -  Release  of  Notice  4 

•  Current  Version  Used  by  Military 

[EtegcssmsIK)  I 


Georgia 
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Applications  of  MIL-STD-1553 


I  I  II 


•  Air  Force 

•  Virtually  Every  Platform 

•  Avionics  and  EW 

•  F-15,  F-16,  A-10,  C-5,  C-130,  C-141 

•  Navy 

•  Surface  and  Subsurface  Ships 

•  Army 

•  Helicopters,  Tanks,  Howitzers 

•  Other 

•  Satellites 

•  Space  Shuttle  Payloads 

•  International  Space  Station 

•  Bay  Area  Rapid  Transit  (BART) 

•  Manufacturing  Production  Lines 


II  I  I  ■ 


Georgia 
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MIL-STD-1553  Basics: 
Overview 


Data  Rate 

1  MHz 

Word  Length 

20  bits 

Data  Bits  /  Word 

1 6  bits 

Message  Length 

Maximum  of  32  Data  Words 

Transmission  Technique 

Half-Duplex 

Operation 

Asynchronous 

Encoding 

Manchester  II  Bi-Phase 

Protocol 

Command  /  Response 

Bus  Control 

One  BC  at  a  Time 

Fault  Tolerance 

Dual  Redundant  (A  &  B  bus) 

Georgia 

Tech 


Lessons  Learned  with  System  Integration  -  20 


MIL-STD-1553  Basics: 
Overview  (Continued) 


Message  Formats 

BC-RT 

RT-BC 

RT-RT 

Broadcast 

System  Control  (Mode  Codes) 

Number  of  Remote  Terminals 

Maximum  of  31 

Terminal  Types 

Remote  Terminal  (RT) 

Bus  Controller  (BC) 

Bus  Monitor  (BM) 

Transmission  Media 

Twisted  Shielded  Pair 

Coupling 

Transformer  and  Direct 

Georgia 

Tech 
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MIL-STD-1 553  Basics: 
Comparison  to  Other  Standards 


100  Kbps  1Mbps  10  Mbps  100  Mbps  1  Gbps  10  Gbps 


Georgia 

Tech 


Lessons  Learned  with  System  Integration  -  22 


Hardware  Upgrades  for  1553 

Integration 


■■ 


Georgia  j I  t^*@@®[r©[]T] 
lech  I  OrosSO&uja® 


Lessons  Learned  with  System  Integration  -  23 


ALR-69  RWR  1553  Upgrade 


•  ALR-69  was  Originally 
Controlled  with  System 
Control  Panels  via  Bi- 
Directional  Discrete 
Signals 

•  The  ALR-69  Class  IV 
Upgrade  Fielded  in  the  Mid 
‘90s  Provided  Two  1553 
Interfaces  for  the  System 

•  The  System  was  Then 
Capable  of  Sharing  Data 
and  Being  Controlled  Over 
1553 


Georgia  j I  t£te@®$[r©[jT] 
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Lessons  Learned  with  System  Integration  -  24 


ALQ-1 31  /ALQ-1 84  ECM  Pods 


•  Both  the  ALQ-1 31  and 
ALQ-1 84  Pods  Were 
Originally  Designed  to 
Interface  with  the  Pulse 
Position  Data  (PPD)  Bus 

•  1553  Interfaces  Have  Been 
Developed  for  Both  Pods, 
But  Have  Not  Been  Used 
Due  to  Limitations 

•  Northrop  Grumman’s 
RePLACE  Hardware 
Solutions  Offer  New  1553 
Interfaces  That  Meet  EW 
Suite  Requirements 


Georgia 

Tech 
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ALQ-213  System  Block  Diagram 

CMSC  (F-16) 


CMSP 


Discretes 


RWR 


Georgia  1 
lech 


1  f  EU 

^  GT 

* 

MS  Disp' 

R I  y  EL! 

1  ■  ' 

SYS  ■  “ 

■  | 

*  — 

11  ws 

SYSTEM  — 

OFF  — 

JTSN  hhi  ■’*' 

■  v 

-  —MODE -  A* 

□,q.  SEMI 

RS-422 


RS-232 


PPD 


Sequencers  /  Dispensers 


RS-485 


ECM  Pod 


RS-422 


AV  Bus  (MIL-STD-1 553) 


EEECP 


n 

EWBus  (MIL-STD-1 553) 

IJ 

□Z  ~n 

SMS 


WMUX 


DVR 


COMET  (A-10  only) 


ALE-50  (F-16  only) 
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Message  Protocol  Schemes 


•  Three  Basic  Types  of  Message  Protocol  Schemes 

*  All  Messages  are  Periodic,  or 

*  All  Messages  are  Aperiodic,  or 

*  Mixture  of  Periodic  and  Aperiodic  Messages 


Georgia 
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Time  Division  Multiplexing 


•  Time  Division  Multiplexing  (TDM)  Is  a  Structured 
Architecture-Level  Protocol  Design  Methodology 

*  All  Messages  are  Periodic 

*  Messages  are  Transmitted  at  Pre-Defined  Rates 

•  Traffic  is  NOT  Sent  on  Basis  of  a  Special  Condition 

•  Fixed  Bandwidth  -  Data  is  Always  on  the  Bus 


Georgia 
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Time  Division  Multiplexing 


•  Advantages 

*  Helps  Ensure  the  Integrity  of  Data  Traffic 

•  Guarantees  Bus  Throughput 

•  Bandwidth  Can  be  Calculated  in  Advance 

•  Fixed  and  Known  Maximum  Latency  for 
Data  Transfers 

*  Disadvantages 

*  Always  Uses  the  Same  Amount  of  Bandwidth 


Georgia 


Lessons  Learned  with  System  Integration  -  29 


System  Control  Over  1553  Interface 


•  The  1553  Interface  Serves  as 
a  Replacement  for  Many  of 
the  Older  Discrete  Signal 
Interfaces 


•  Every  Single  Control 
Function  Should  Have  an 
Associated  Response 

•  This  is  Typically 
Accomplished  Using  a 
Control  Message  and  a 
Status  Message 


System 


Subsystem 


Georgia  j I 
lech  || 


Lessons  Learned  with  System  Integration  -  30 


System  Status  Message 


•  Overall  Health  of  the  System 

•  Real-Time  State  of  the  System  (i.e.  Warm-up,  Test 
Mode,  Power  Down  Mode,  etc.) 

•  Used  for  Mission  Critical  &  Non-Mission  Critical 
Failure  Determination 

•  Detailed  Modules’  Status  is  Rolled  Up  into  Overall 
System  State 
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System  Status  Roll-up 


•  Overall  System  Health 

•  Needed  by  Pilots,  Engineers,  & 
Other  Systems 


•  Health  &  State  of  System  Modules 

•  Needed  by  Pilots,  Engineers,  & 
Other  Systems 


•  Detailed  Status  of  System  Modules 

•  Used  for  Post  Analysis  by  System 
Experts 
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Header/Record  Data  Formats 


•  Used  When  There 
are  Multiple  Entries 
of  Data  with  the 
Same  Format 

•  Commonly  Used 
with  Track  Files 

•  Frequency  of  Header 
is  Proportional  to 
the  Number  & 
Frequency  of 
Records 


Header 
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Message  Overloading 


Overloading  is  Used 
When: 

*  Limited  Number  of 
Subaddresses  are 
Available 

*  Information  with  Similar 
Data  Formats 

Advantage:  Reduces 
Overall  Bandwidth 
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Instrumentation  Messages 


*  Typically  Assigned  to  1  Subaddress 


•  Message  Overloading  May  be 
Necessary 

•  Usually  Sent  at  Fastest  Rate  Possible 

•  Contents  of  Instrumentation  Messages 
Should  Not  Be  Used  by  Other  Systems 
for  Integration  Purposes 

•  Data  Used  by  System  Experts  for 
Trouble  Shooting  and  Post  Analysis 
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Extending  the  Standard 


•  Zero-Fill  Unused  Data 

•  Attempt  to  Have  “0”  as  an  Undefined  Value 

•  Subaddress  Assignment 

•  Unassigned  Subaddresses  Should  Always  Respond  With  Zero- 
Filled  Data 

•  Similar  Systems  Should  Have  Similar  Subaddress  Assignments 

•  Multiple  Busses  on  an  Aircraft  Should  be  Synchronized 

•  Accomplished  via  Timing  Messages  &  1553  Mode  Codes 
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Benefits  of  1553  Integration 

•  Less  Hardware  &  Software  Support 

•  Discrete  Signal  &  Serial  Interfaces  are  Replaced 

•  Migration  Towards  an  Integrated  Suite 

•  Mission  Data  &  Techniques  Can  be  Coordinated 

•  Systems  Can  Share  Information 

•  Everybody  Speaks  the  Same  Language 

•  Protocol  Governs  What  Can  and  Cannot  be  Done 

•  Format  of  Documentation  is  Standardized 

•  Reduction  in  System  Integration  Time  &  Effort 
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Questions? 


■■ 
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Lessons  Learned  with 
Integration  of  Federated 
Systems  on  Legacy 
Aircraft 


Adam  McCorkle 
GTRI/ELSYS 

adam.mccorkle@gtri.gatech.edu 


Kendal  Hinton 
GTRI/ELSYS 

kendal.hinton@gtri.gatech.edu 
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Phone:  404-407-6427 


Phone:  404-407-6042 


F-16C+  Fighting  Falcon 


•  F-16C+ 

•  F-1 6  Preblock  40 

•  Blocks  25,  30,  and  32 

•  Approximately  600  Active  Aircraft 

•  ANG 

•  AFRC 

•  ACC 

•  EW  Suite 

•  ALQ-21 3  Countermeasures  Set  (CMS) 

•  ALR-69  Radar  Warning  Receiver  (RWR) 

•  Electronic  Countermeasures  (ECM)  Pods 

•  ALQ-131 

•  ALQ-184 

•  ALE-47  Countermeasures  Dispenser  System  (CMDS) 

•  Partial 

•  Theater  Airborne  Reconnaissance  System  (TARS) 

•  ALE-50  Towed  Decoy 
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A-10  Thunderbolt 


•  A-10A 

•  “Warthog” 

•  A-10C 

•  Upgrade  to  Avionics,  Weapons,  and  Cockpit  Display  Systems 

•  Approximately  400  Active  aircraft 

•  ACC 

•  AFRC 

•  ANG 

•  EW  Suite 

•  ALQ-213  CMS 

•  ALR-69  RWR 

•  ECMPods 

•  ALQ-131 

•  ALQ-184 

•  AAR-47  Missile  Warning  System  (MWS) 

•  ALE-47  CMDS 

•  Comet  IR  Dispense  System 
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ALQ-213  Countermeasures  Set  (CMS) 


•  Replaces  All  Existing 
Discrete  EW  Control  Panels 

•  Single  Point  of  Control  of  All 
EW  Systems 

•  Semi-Automatic  and 
Automatic  Capabilities  That 
Adapt  to  Threat  Environment 
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II  I  1 1 II 


ALR-69  Radar  Warning  Receiver 

(RWR) 

•  Purpose  -  Detect,  Identify, 
and  Output  Visual/Audio 
Information  for  Friendly 
and  Hostile  RF  Radar 
Emitters 

•  Quadrant  Detection 

•  Visual  Output  of 
Symbology  and  Angle  of 
Arrival 

•  Audio  Output  via  Aircraft 
Intercom 
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Electronic  Countermeasures  (ECM) 

Pods/Jammers 


•  Purpose  -  Provide 
Defensive  Protection  for 
Aircraft  Through  Defeat 
or  Deception  of  Radar 
Systems  Attempting  to 
Track  the  Weapons 
Platform 

•  ALQ-131 

•  ALQ-184 


ALQ-131  ECM  Pod 
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I  I  I  I  II 


ALE-47  Countermeasure  Dispense 

System 


•  Purpose  -  Provide 
Defensive  Protection  for 
the  Weapons  Platform 
Through  Ejection  of 
Materials  Designed  to 
Distract  or  Decoy  RF  and 
IR  Based  Systems. 


•  Chaff  -  Counters  RF- 
Based  Tracking 
Systems 
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•  Flares  -  Counters  IR- 
Based  Tracking 
Systems 
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AAR-47  Missile  Warning  System  (MWS) 


•  Purpose  -  Detect  and 
Identify  IR  Emissions  Near 
the  Weapons  Platform, 
and  Output  Visual 
Information  to  the 
Aircrew. 

•  Automatically  Triggers  an 
Appropriate 
Countermeasure  to 
Protect  the  Pilot  and 
Aircraft 
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Theatre  Airborne  Reconnaissance 

System  (TARS) 


•  Purpose  -  Provide  Under- 
the-Weather,  Medium  to 
High  Threat,  Daytime 
Imagery  Collection 

•  Tactical  Reconnaissance 
Pod  System 
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ALE-50  T owed  Decoy 


■■■ 


•  Purpose  -  Provide  a 
Radar  Target  Decoy  for  an 
Incoming  Missile 

*  Towed  Behind  the  Aircraft 
When  Deployed  and 
Protects  the  Aircraft 
Against  RF-Homing 
Missiles  by  Drawing  Them 
Away  From  the  Host 
Aircraft 
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COMET  Dispenser  Pod 


•  Purpose  -  Provides 
Extended  Duration 
Protection  for  Aircraft  in 
Infrared  Threat 
Environment 

•  MIL-STD-1553  Control 
Interface 

•  Developmental  System 
Not  Currently  Fielded 
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History  of  System  Integration 


•  1950s  and  1960s  -  Aviation  Electronics  are  Mainly 
Analog  and  Stand-alone  Systems 

•  Late  1960s  and  1970s  -  Became  Necessary  to  Share 
Information  Between  Systems 

*  Accomplished  Primarily  via  Discrete  Analog  Signal 
Interfaces  and/or  A/D  Converters 

*  Required  Increasingly  Complex  Wiring  and 
Connections 
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History  of  System  Integration 


•  Late  1970s  -  Advent  of  Digital  Technology  Made 
Possible  Increasingly  Sophisticated  Systems 
Powered  by  Computers 

*  Led  to  a  Need  for  Higher  Speed  Transfer  of  Data 
Between  Systems 

*  Digital  Interfaces  Conceived  to  Address  this  Problem 
But  With  an  Increase  in  the  Amount  of  Wiring  and 
Connector  Complexity  Required 

*  Eventually  the  Need  Arose  for  a  Transmission  Medium 
that  Would  Allow  Multiple  Systems  to  Share  a  Single, 
Common  Set  of  Wires 
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Purpose  of  MIL-STD-1553 


•  MIL-STD-1553  Was  Developed  to  Provide  a  Standard 
Solution  that  Would  Allow  Multiple  Systems  to 
Share  a  Single  Common  Set  of  Wires  (Data  Bus)  in 
Order  to  Efficiently  Move  Data  Between  Systems 
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Purpose  of  MIL-STD-1553 

Discrete  Analog  Signals 


Purpose  of  MIL-STD-1553 
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History  of  MIL-STD-1553 


*  Late  1960’s  -  H-009  Data  Bus  Specification  Used  on 
the  F-15  (Early  Forerunner  of  1553) 

*  1968  -  SAE  Establishes  Subcommittee  to  Define  a 
New  Serial  Bus  Standard  for  the  Military  Avionics 
Community 

•  1970  -  First  Draft  (From  SAE/A2K)  Developed 

•  1973  -  MIL-STD-1553  Released 

•  USAF  Only  Version  of  the  Standard 

•  Primary  User  was  the  F-16 
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History  of  MIL-STD-1553 

•  1975  -MIL-STD-1553A  Released 

•  First  Tri-Service  Version  of  the  Standard 

•  F-16 

•  AH-64A 

•  1978  -  MIL-STD-1553B  Released 

•  Standard  “Frozen”  at  Revision  “B” 

•  Allowed  for  Component  Manufacturers  to  Develop  Standard 
Products  That  Provide  for  Easy  Implementation 

•  1996  -  Release  of  Notice  4 

•  Current  Version  Used  by  Military 

[EtegcssmsIK)  I 
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Applications  of  MIL-STD-1553 


I  I  II 


•  Air  Force 

•  Virtually  Every  Platform 

•  Avionics  and  EW 

•  F-15,  F-16,  A-10,  C-5,  C-130,  C-141 

•  Navy 

•  Surface  and  Subsurface  Ships 

•  Army 

•  Helicopters,  Tanks,  Howitzers 

•  Other 

•  Satellites 

•  Space  Shuttle  Payloads 

•  International  Space  Station 

•  Bay  Area  Rapid  Transit  (BART) 

•  Manufacturing  Production  Lines 


II  I  I  ■ 
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MIL-STD-1553  Basics: 
Overview 


Data  Rate 

1  MHz 

Word  Length 

20  bits 

Data  Bits  /  Word 

1 6  bits 

Message  Length 

Maximum  of  32  Data  Words 

Transmission  Technique 

Half-Duplex 

Operation 

Asynchronous 

Encoding 

Manchester  II  Bi-Phase 

Protocol 

Command  /  Response 

Bus  Control 

One  BC  at  a  Time 

Fault  Tolerance 

Dual  Redundant  (A  &  B  bus) 
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MIL-STD-1553  Basics: 
Overview  (Continued) 


Message  Formats 

BC-RT 

RT-BC 

RT-RT 

Broadcast 

System  Control  (Mode  Codes) 

Number  of  Remote  Terminals 

Maximum  of  31 

Terminal  Types 

Remote  Terminal  (RT) 

Bus  Controller  (BC) 

Bus  Monitor  (BM) 

Transmission  Media 

Twisted  Shielded  Pair 

Coupling 

Transformer  and  Direct 
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MIL-STD-1 553  Basics: 
Comparison  to  Other  Standards 


100  Kbps  1Mbps  10  Mbps  100  Mbps  1  Gbps  10  Gbps 


Georgia 

Tech 


Lessons  Learned  with  System  Integration  -  22 


Hardware  Upgrades  for  1553 

Integration 


■■ 
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Diminishing  Manufacturing  Sources 


•  GTRI  Developed  SUSTAIN  Tool 
Helps  Catalog  High  Risk  DMS 
Components 

•  SUSTAIN  Reports  From  System 
Level  Down  to  Lowest  Level 
Assemblies  (LLA) 

•  Concerned  With: 

•  High  Number  of  Mission 
Incapable  Supplies  (MICAPS) 

•  Low  Mean  Time  Between 
Failures  (MTBFs) 

•  High  Cost  Components 

•  Obsolete  Components 

•  Spare  Assemblies  in  Stock 

•  SUSTAIN  Predicts  Mission 
Degradation  Based  on  These 
Factors 


Mission  Degradation  for  318470-000 


[7  3 1 8470-000  □  3 1 8470-000  (LLfiJ)  P  0  S  Ajthorized  [~  Q  3 1 8470-000  (LLA)  WRM  Ajthorized 

|“  Q  318470-000  (HLfi)  POS  Ajthorized  |“  ^  318470-000  (HLA)  WRM  Ajthorized 

1 1  Total  Spares:  1  from  Current  Asby,  0  Cannibalized! 
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ALR-69  RWR  1553  Upgrade 


•  ALR-69  was  Originally 
Controlled  with  System 
Control  Panels  via  Bi- 
Directional  Discrete 
Signals 

•  The  ALR-69  Class  IV 
Upgrade  Fielded  in  the  Mid 
‘90s  Provided  Two  1553 
Interfaces  for  the  System 

•  The  System  was  Then 
Capable  of  Sharing  Data 
and  Being  Controlled  Over 
1553 
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ALQ-1 31  /ALQ-1 84  ECM  Pods 


•  Both  the  ALQ-1 31  and 
ALQ-1 84  Pods  Were 
Originally  Designed  to 
Interface  with  the  Pulse 
Position  Data  (PPD)  Bus 

•  1553  Interfaces  Have  Been 
Developed  for  Both  Pods, 
But  Have  Not  Been  Used 
Due  to  Limitations 

•  Northrop  Grumman’s 
RePLACE  Hardware 
Solutions  Offer  New  1553 
Interfaces  That  Meet  EW 
Suite  Requirements 
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ALQ-213  System  Block  Diagram 

CMSC  (F-16) 


CMSP 


Discretes 


RWR 
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Message  Protocol  Schemes 


•  Three  Basic  Types  of  Message  Protocol  Schemes 

*  All  Messages  are  Periodic,  or 

*  All  Messages  are  Aperiodic,  or 

*  Mixture  of  Periodic  and  Aperiodic  Messages 
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Time  Division  Multiplexing 


•  Time  Division  Multiplexing  (TDM)  Is  a  Structured 
Architecture-Level  Protocol  Design  Methodology 

*  All  Messages  are  Periodic 

*  Messages  are  Transmitted  at  Pre-Defined  Rates 

•  Traffic  is  NOT  Sent  on  Basis  of  a  Special  Condition 

•  Fixed  Bandwidth  -  Data  is  Always  on  the  Bus 
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Time  Division  Multiplexing 


•  Advantages 

*  Helps  Ensure  the  Integrity  of  Data  Traffic 

•  Guarantees  Bus  Throughput 

•  Bandwidth  Can  be  Calculated  in  Advance 

•  Fixed  and  Known  Maximum  Latency  for 
Data  Transfers 

*  Disadvantages 

*  Always  Uses  the  Same  Amount  of  Bandwidth 
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System  Control  Over  1553  Interface 


•  The  1553  Interface  Serves  as 
a  Replacement  for  Many  of 
the  Older  Discrete  Signal 
Interfaces 


•  Every  Single  Control 
Function  Should  Have  an 
Associated  Response 

•  This  is  Typically 
Accomplished  Using  a 
Control  Message  and  a 
Status  Message 


System 


Subsystem 
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System  Status  Message 


•  Overall  Health  of  the  System 

•  Real-Time  State  of  the  System  (i.e.  Warm-up,  Test 
Mode,  Power  Down  Mode,  etc.) 

•  Used  for  Mission  Critical  &  Non-Mission  Critical 
Failure  Determination 

•  Detailed  Modules’  Status  is  Rolled  Up  into  Overall 
System  State 
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System  Status  Roll-up 


•  Overall  System  Health 

•  Needed  by  Pilots,  Engineers,  & 
Other  Systems 


•  Health  &  State  of  System  Modules 

•  Needed  by  Pilots,  Engineers,  & 
Other  Systems 


•  Detailed  Status  of  System  Modules 

•  Used  for  Post  Analysis  by  System 
Experts 
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Header/Record  Data  Formats 


•  Used  When  There 
are  Multiple  Entries 
of  Data  with  the 
Same  Format 

•  Commonly  Used 
with  Track  Files 

•  Frequency  of  Header 
is  Proportional  to 
the  Number  & 
Frequency  of 
Records 


Header 
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Message  Overloading 


Overloading  is  Used 
When: 

*  Limited  Number  of 
Subaddresses  are 
Available 

*  Information  with  Similar 
Data  Formats 

Advantage:  Reduces 
Overall  Bandwidth 
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Instrumentation  Messages 


*  Typically  Assigned  to  1  Subaddress 


•  Message  Overloading  May  be 
Necessary 

•  Usually  Sent  at  Fastest  Rate  Possible 

•  Contents  of  Instrumentation  Messages 
Should  Not  Be  Used  by  Other  Systems 
for  Integration  Purposes 

•  Data  Used  by  System  Experts  for 
Trouble  Shooting  and  Post  Analysis 
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Extending  the  Standard 


•  Zero-Fill  Unused  Data 

•  Attempt  to  Have  “0”  as  an  Undefined  Value 

•  Subaddress  Assignment 

•  Unassigned  Subaddresses  Should  Always  Respond  With  Zero- 
Filled  Data 

•  Similar  Systems  Should  Have  Similar  Subaddress  Assignments 

•  Multiple  Busses  on  an  Aircraft  Should  be  Synchronized 

•  Accomplished  via  Timing  Messages  &  1553  Mode  Codes 
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Benefits  of  1553  Integration 

•  Less  Hardware  &  Software  Support 

•  Discrete  Signal  &  Serial  Interfaces  are  Replaced 

•  Migration  Towards  an  Integrated  Suite 

•  Mission  Data  &  Techniques  Can  be  Coordinated 

•  Systems  Can  Share  Information 

•  Everybody  Speaks  the  Same  Language 

•  Protocol  Governs  What  Can  and  Cannot  be  Done 

•  Format  of  Documentation  is  Standardized 

•  Reduction  in  System  Integration  Time  &  Effort 
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Questions? 


■■ 


DIS^ 

Defense  Information  Systems  Agency 

Department  of  Defense 


A  Strategy  for  Program  Success: 
Leading  Indicators  for  Early 

Warning 

DISA  Office  of  the  CAE 
Ms  Rebecca  Cowen-Hirsch 

703-882-2533 

Rebecca.Cowen-Hirsch@disa.mil 

25  October  2006 


UNCLASSIFIED 


MSA- 


Agenda 


*  Context 

*  Purpose  of  Briefing 

*  A  Framework  for  Indicators 

*  Typical  Indicators  of  Success  and  Problems 

*  Leading  Indicators  and  What  They  Offer 

-  Some  Examples 

*  Case  Studies 

*  Future  Work 


MSA- 


Context  - 1 


Context  -  2 


•  DISA’s  systems  engineering  &  acquisition  foci  have  driven 
this  research 

-  Communications  systems  and  managed  services  acquisition 

-  Information  systems  infrastructure  development  and  managed 
services  acquisition 

-  Information  systems  C2  applications  development  and 
managed  services  acquisition 

•  Other  organizations’  systems  engineering  &  acquisition 
environments  may  require  different  or  additional  leading 
indicators 

-  Weapons  platform  hardware  development 

-  Weapons  subsystem  hardware  development 

-  Aerospace  platform  hardware  development 

-  Aerospace  subsystem  hardware  development 

-  Hard-real-time  software  development 
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Purpose 


*  Track  major  programs  from  acquisition  & 
systems  engineering  perspectives  as  part  of 
responsible  technical  and  program  management 

-  Using  conventional  metrics 

*  Often  lagging  indicators 

-  Using  unconventional  indicators 

•  Early  warning  system 

*  Assess  indicators  over  time  to  capture  lessons 
and  refine  indicators  for  early  warning  of 
problems 


About  Leading  Indicators 


*  Leading  indicators  give  a  sense  of  problems 
downstream  in  a  system  or  service  acquisition  or 
development 

*  These  indicators  help  a  program  manager  or  chief 
engineer  identify  potential  problems  earlier  than 
otherwise 

*  Once  identified,  problems  may  be  amenable  to 
more  traditional  risk  management  and 
remediation 

*  Leading  Indicators  may  first  appear  subtly  rather 
than  as  a  waving  red  flag 
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about  Leading  Indicators 


*  Indicators  need  to  identify  problems  that  have 
tractable  solutions 

-  Confirming  a  future  wreck  that  cannot  be  avoided  is  only  so 
valuable 

*  Indicators  should  not  require  yet  another  data  call 

-  If  a  simple  “yes,”  “no,”  or  “the  answer  off  the  top  of  my  head 
is  6”  response  to  a  simple  question  isn’t  possible  without 
gathering  further  data,  then  we  have  the  wrong  indicator 

*  Indicators  should  be  readily  understandable  by  all 

-  If  it  needs  an  explanation,  it’s  a  complex  metric,  not  an 
indicator 
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DlS/^he  Program  Manager’s  Challenge 


Calendar-driven 
Two-year  cycle 


Threat-driven 
One+  year  cycle 


Event-driven 
One+  year  cycle 


Source:  NDIA-NAU  Course  in  Defense  Systems  Acquisition  Management 


These  DoD  processes  should  interact  &  overlap. 

The  program  manager  must  orchestrate  across  all  three. 


DIS^a  Framework  for  Indicators 


Program  Factors 

Examples  of  Indicators 

Program 

Management 

Degree  of  commitment  to  program  from  other  relevant  organizations 

Degree  of  program  management  engagement  in  the  program 

Communications 

Cordiality,  frequency  of  communications  between  program  personnel  and 
stakeholders 

Speed  of  stakeholder  issues  resolution 

Fiscal  Issues 

Degree  of  agreement  among  stakeholders  on  funding  approach  &  commitments 

Fiscal  unanimity  among  key  organizations  (e.g.,  Congress,  OSD,  Military  Services) 

Requirements 

Stability  of  requirements 

Realism  of  requirements  versus  maturity  of  technology  to  meet  them 

Schedule 

Schedule  stability 

Evidence  of  unrealistic  scheduling 

Staffing 

Government  /  contractor  management  stability,  lack  of  gaps  in  management 

Key  government  /  contractor  staff  stability 

Technology 

Technology  readiness  to  be  employed  in  DoD  systems 

Technological  skills  available  in  government  and  contractor  base 

ws/V  Framework  (concluded) 


*  Dimensions  of  the  indicator  framework 

-  Stage  in  the  system  or  service  life  cycle 

•  Some  indicators  are  more  significant  for  some  milestones 
than  others 

-  E.g.,  technology  can  be  less  mature  prior  to  Milestone  A  than 
prior  to  Milestone  C 

-  Duration  of  an  indicator  problem 

•  Some  indicators  show  more  sensitivity  to  duration  of  an 
indicated  problem  than  others 

-  E.g.,  a  gap  in  a  key  government  management  position  of  any 
duration  can  portend  downstream  problems  in  a  program 
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qiqt  Indicators  &  Their  Relevance  in  the 
F  System  or  Service  Life  Cycle 


A 

A 

A 

IOC 

FOC 

System 

Integration 

System  LRIP 

Demonstration 

Full-Rate  Prod 
&  Deployment 

Sustainment 

Disposal 

/\  Concept 

V  Decision 

< 

Design 

Readiness 

Review 

FRP 

Decision 

Review 

Concept  Technology 

Refinement  Development 

System  Development 

&  Demonstration  Production  &  Deployment 

Operations 
&  Support 

Pre-Systems  Systems  Acquisition  Sustainment 


Acquisition 


BIS£^What  Indicators  Can  Indicate 


Indicator 


Possible  Consequences 


Reduction  in  face-to-face 
communications 


Lack  of  realism  regarding 
financial  constraints 


Can  result  in  failure  to  resolve  stakeholder  issues  that  can  get 
pushed  aside  and  reappear  later  in  the  process  when  correction  costs 
more.  In  extreme  cases,  can  lead  to  funding  cuts  through  lack  of 
advocacy  by  key  stakeholders. 

Most  likely  will  result  in  significant  cost  overruns  if  the  parties  do  not 
understand  the  nature  of  contract  funding.  Can  result  in  IG  and  GAO 
involvement  for  significant  overrun. 


Requirements  scope  creep  Generally  will  result  in  cost  overruns  or  incomplete  capability  being 

delivered,  and  will  also  cause  issues  regarding  contract  compliance 
between  government  and  contractor.  Can  also  result  in  IG  and  GAO 
oversight  for  significant  overrun. 


Unrealistic  schedule  Most  likely  will  result  in  a  program  off  the  critical  path  from  the  start. 

Can  affect  other  programs  under  development  (via  dependencies). 
Can  affect  costs  if  the  contractor  has  to  add  more  personnel  to  try  to 
get  a  program  back  on  track.  Quality  of  delivered  product  or  service 
may  be  reduced  while  trying  to  meet  schedule. 

Technology  not  yet  ready  Can  result  in  failure  to  deliver  required  product  or  service,  or  at 

least  will  result  in  delayed  delivery. 
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Indicator 

Reduction  in  face  to  face 
communications 

Lack  of  realism  regarding 
financial  constraints 

Requirements  scope  creep 

Unrealistic  Schedule 

Technology  not  yet  ready 


Correcting  the  Problem 


Possible  Solutions 


Primary  solution  is  to  identify  the  cause  of  the  reduction  in 
communications.  If  necessary  set  mandatory  meetings  with  posted 
agenda.  Have  senior  management  attend  the  meetings  to  oversee 
interaction  between  parties. 

Proper  training  can  usually  correct  this  problem.  A  closer  scrutiny  of 
spending  and  program  budget  may  be  required  by  senior 
management.  Have  PM  explain  5%  or  great  variance. 

A  watchful  contracting  officer  should  help  significantly  to  reduce 
requirements  creep.  Educating  the  COTR  can  also  aid  in  reducing 
creep. 


This  problem  is  easier  to  prevent  that  correct.  Upon  identification, 
meet  with  team  to  define  realistic  schedule  and  re-baseline  to  that 
schedule.  Costs  may  be  affected. 


Identify  less  risky  technology,  experiment  with  it,  pilot  it, 
demonstrate  how  well  it  scales.  Alternatively,  pilot  the  riskier 
technology,  work  out  the  bugs  and  deliver  in  smaller  increments,  if 
feasible.  Either  solution  results  in  schedule  delays,  possible  ^ 
capability  reduction. 


Study  1 :  The  Problems 


Context:  complex  suite  of  information  services;  many 
diverse  users,  DoD-wide;  some  applications  services,  some 
infrastructure  services 


Leading  indicators  getting  to  Milestones  A  &  B 
v^Program  management,  Staffing 


•  Leadership  (government,  contractor)  absence,  turnover,  weak 
leadership 

yr  •  Key  technical  staff  (government,  contractor)  absence,  turnover 

^  Communication 


•  Turnover  of  key  stakeholder  caused  schedule  slip  while  working 
-  to  win  over  the  replacement  stakeholder 
^ Fiscal  issues 


•  Disagreement  among  stakeholders  on  program  funding 
J  ( Requirements 
\r  Schedule 


•  Realistic  until  personnel  churn  caused  program  to  lose  focus 

^Technology  maturity,  skills  to  use 

•  Technology  not  yet  widely  used  in  commercial  world 

•  Lack  of  available  skills  in  government,  among  government 
contractors 
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DIS/^Qase 

Study  1:  The  Resolution* 


•  Context:  complex  suite  of  information  services;  many  diverse 
users,  DoD-wide;  some  applications  services,  some 
infrastructure  services 

•  Leading  indicators  getting  to  Milestones  A  &  B 
^Program  management,  Staffing 

•  Leadership  (government,  contractor)  -  gained  longer-term  commitments 
from  new  talent,  stabilized  situation 

•  Key  technical  staff  (government,  contractor)  -  attracted  talented  new 
players  with  commitment  to  complete  the  first  spiral  at  least 

v'  Communication 

•  Turnover  of  key  stakeholder  -  still  building  trust  via  intensive 
communication  campaign 

Fiscal  issues 

among  stakeholders  -  still  a  challenge 


•  Realistic  until  personnel  churn  -  working  hard  but  with  possible  slip 

Technology  maturity,  skills  to  use 

•  Technology  not  yet  widely  used  -  still  experimenting;  outcome  not  clear 

•  Lack  of  skills  on  program  -  OJT,  experience  building  but  likely  schedule 

slips  downstream  15 

*So  far ... 


•  Disagreement 
JA  Requirements  - 
^  Schedule 


Study  2:  The  Problems 


*  Context:  worldwide  communications  system 
ground-based  infrastructure,  being  developed  & 
fielded  in  multiple  spirals 

*  Leading  indicators  getting  to  MSs  A  &  B  of  Spiral  N 

QjjgProgram  management,  Staffing 

S  Communication 

•  Multiple  Military  Service  stakeholders  have  proven  challenging 
for  a  small-staff  PMO 

J ^Fiscal  issues 
^Requirements 

•  One  key  requirement  dependent  on  immature  technology 

^Schedule 

•  Realistic  except  for  immature  technology,  errant  stakeholders 

^ Technology  maturity 

•  One  key  product  still  in  development 

•  Reliant  on  multiple  stakeholders  for  agreement  on  16 

technological  way-ahead 


DIS/^Qase 

Study  2:  The  Resolution* 


•  Context:  worldwide  communications  system  ground-based 
infrastructure,  being  developed  &  fielded  in  multiple  spirals 

•  Leading  indicators  getting  to  MSs  A  &  B  of  Spiral  N 
J^Program  management,  Staffing 

Communication 

•  Multiple  Military  Service  stakeholders  -  focused  on  most  challenging 
player(s) 

Q]l£  ^ Fiscal  issues  -  cost  increase  due  to  delayed  requirement 

^Requirements 

•  Dependent  on  immature  technology  -  delaying  fielding  of  new  technology; 
extending  life  of  older  technology  by  two  years 

v^Schedule 

•  Immature  technology,  errant  stakeholders  -  generally  acceptable  schedule 
slip  accorded  to  technology  problem 

^Technology  maturity 

•  Key  product  in  development  -  too  early  to  tell  if  new  schedule  realistic 

•  Reliant  on  multiple  stakeholders  for  agreement  -  achieved  grudging 
agreement 
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*So  far ... 


MSA- 


Summary 


Source:  NDIA-NAU  Course  in  Defense  Systems  Acquisition  Management 


Most  of  the  leading  indicators  address  Defense  Acquisition, 
but  without  acting  to  correct  indicated  problems,  damaging 

effects  can  occur  in  all  three  domains 


Future  Work 


*  Refine  number  and  variety  of  leading  indicators 

-  Most  significant  effects  likely  on  program 

-  Modest  data  collection  demands 

*  Gather  data  across  a  greater  variety  of  programs 

-  Hardware-intensive 

-  Software-intensive 

-  Leading-edge  technology  application 

-  Complex  dependencies  on  programs  inside,  outside  the 
Agency 

*  Analyze  results 

*  Refine  indicators 

*  Define  dashboard  with  watch  list  of  indicators 
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MSA- 


Questions? 
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Backup  Slides 
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□ISA  Candidate  Early  Warning  Indicators  - 1 


Program  Advocacy 

•  Weak/Ineffective  Program  Manager 

•  Lack  of  Management  Continuity  and  Oversight 

•  Lack  of  Involvement  from  Stakeholders  (contracting  officers, 
COTRs,  OSD,  etc.) 

•  Reluctant  or  ambiguous  statements  from  public  officials 

•  Programs  not  clearly  specified  in  budget  submissions,  FYDP, 
Program  Element  Descriptions  Summary  (PEDS),  etc. 

Communications 


•  Principals  do  not  attend  regular  meetings  and  do  not  send  an 
alternate 

•  Reduction  in  face-to-face  communications 

•  Inadequate  resolution  of  issues 

•  Prime  contractor  not  engaged 

•  Lack  of  proactive  stakeholder  communication 

•  Lack  of  information  sharing 

•  Lack  of  visibility  on  executive  status  or  priority  list 
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□ISA  Candidate  Early  Warning  Indicators  -  2 


Fiscal  Issues 


*  Inability  to  tie  performance  to  well  articulated  payoff 

*  Lack  of  realism  regarding  fiscal  constraints  at  time  of 
contract  award 

*  Submission  of  multiple  engineering  change  proposals 
(ECPs) 

*  Large  underruns  or  overruns  inconsistent  with 
contractor  proposal  within  three  months  of  award 

*  Military  Services/Components  disagreeing  on  funding 
approach  or  commitments 

*  Contractor  proposal  estimates  significantly  exceed  IGCE 
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□ISA  Candidate  Early  Warning  Indicators  -  3 


Requirements 

*  Ambiguity  of  requirements 

*  Unrealistic  requirements 

*  Requirements  not  testable 

*  Scope  creep 

*  Mission  needs/requirements  not  adequately  defined  in 
Statement  of  Work 

*  Key  DoD  5000  requirements  documents  not  developed, 
approved,  or  updated 
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mS/.  ^Candidate  Early  Warning  Indicators  -  4 

Schedule 


*  Unrealistic  schedules 

*  Unilaterally  set  schedules 

*  Frequent  schedule  slips  -  especially  w/in  first  3-6  months 

*  Poor  or  no  design/test  schedules 

*  Integrated  Master  Schedule  not  linked  with  contractual  delivery 
schedule 

Staffing 

*  Turnover  ratios  greater  than  5%,  especially  among  staff  with  scarce 
skills 

*  Changes  to  contractor  staffing  within  first  six  months  not  identified  in 
contractor  proposal 

*  Change  in  program  management  of  contractor  or  government 

*  Time  gaps  between  managers 

*  Oversight  officials  lacking  proper  certification 

*  Contracting  officer  not  assigned  or  insufficient  contracting  office 

support  25 


□ISA  Candidate  Early  Warning  Indicators  -  5 


Technology 

*  Proposed  technology  is  below  Technology  Readiness 
Level  of  6  at  Milestone  B 

*  Technology  has  not  been  or  has  been  only  rarely 
demonstrated  successfully  in  commercial  world 

*  Skills  among  government  and  contractor  teams 
insufficient  to  apply  chosen  technology 

Other? 


•  Program  dependencies  -  on  other  programs;  others 
dependent  on  it;  on  outside-the-Agency  organizations 
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□IS/V.A  Framework  for  Indicators(l) 


Program 

Resources 

-Stability  of  Mgrs 
-Stability  of  Staff 
-Financial  Status 


Program 

Communications 
-Degree  of  open  comms 
with  key  stakeholders 
-Degree  of  open  comms 
with  internal  players 


Contractor  mgt 
changed  once 
over  past  year 


*  OSD  stakeholders 
not  being  kept 
current  on  activities 


Program 

Execution 
-  Requirements 
-Schedule 
-Technical  Maturity 


Watch/ 
Take  action 

Neutral/ 

No  action 
required 


Program 

Dependencies 

-On  Other  Progi¥ms 
-On  Other  Organizations 


DIS/!^ 


A  Framework  for  lndicators(2):  Enterprise 

Systems  Engineering  Profiler™ 


Messy  Frontier 

Transitional  Domain 

Traditional  Program 


Sector 


Indicators 


Mission  Environment  Requirements  Stability 

Schedule  Realism 


Scope  of  Effort 


Requirements  Realism 


Scale  of  Effort 


Technology  Readiness? 


Acq.  Environment 


System  Dependencies 
Organizational  Dependencies 


Stakeholder  Involvement  Communication  w/Stakeholders 

-  Degree  of  Agreement,  Advocacy 

-  Frequency  of  Communication 
Stakeholder  Rel’nship  Communication  w/Stakeholders 

-  Cordiality 


Desired  Outcome 


Technology  Readiness 


System  Behavior 


Technology  Readiness? 


Important  Indicators  not  Fitting  Framework 

Management  Stability 

-  Government 

-  Contractor 
Staff  Stability 
-Government 

-  Contractor 

Fiscal  Unity?  28 
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“The  Net-Centric  Foxhole ” 
Perspectives  from  Army 
Future  Combat  Systems 


* 


m 


25  OCT  06 
D.  Emery, 

LTC  D.  Bassett, 

LTC  M.  Schnaidt 
Product  Manager, 

(BCT)  Software  Integration 


Approved  for  Public  Release,  Case  06-6147  (PM  FCS  (BCT) 


Outline 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


•FCS  Program  Overview 

•FCS  “Distributed  System”  architecture  and  Net-Centric  Concerns 
•Issues  from  each  of  the  4  Net-Centric  Checklist  areas 
-Transport 
-IA 

-  Core  Services 

-  Data  &  Applications 

'FCS  approach  to  the  Net  Ready  KPP 
•  Lessons  Learned 
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/*/?0  GRAM  n/iJ\  i\M\GI=  /? 


FCS  System-of-Systems  (SoS) 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


Ba'ttlenJQm  mand 
. SOSCOE,,,,,, 
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Vehicle  (ICV) 


Unmanned  Aerial  Vehicles 
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Mounted  Combat 
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System^ 


Manned 
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ommand  and 
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Class  I 
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Intelligent 
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Systems 
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Surveillance  Vehicle  (RSV) 
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Non-Line  of  Sight 
Cannon  (NLOS-C 


Non-Line  of  Sight  Mortar  (NLOS-M) 


Small 

(Manpackable)  UGV 


ARV 


ARV  RSTA 


Medical 
Treatment  (MV-T) 


FCS  Recovery  and 
Maintenance 
Vehicle  (FRMV) 


Armed  Robotic 
Vehicle  (ARV) 

ical  Vehicle 

1  Evacuation  (MV-E) 
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ARV-A  (L) 


MULE  MULE 

(Countermine)  (Transport) 


8-1-06  1 
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FCS  Unit  of  Action  Elements 

Software  resides  in  all  Prime  Items 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


Manned  Ground  Vehicles 


Infantry 


Mounted 


NLOS  Cannon 


Maint.  &  Recovery 


Command  &  Control 


Medical 


Approved  for  Public  Release:  PM  FCS  (BCT)  case  06-6147 


C4ISR 

Battle  Command  &  Mission  Execution  -  Raytheon 
SOSCOE  /  Warfighter  Machine  Interface  -  Boeing 
Level  1  Fusion  -  Lockheed  Martin 
Sensor  Data  Mgmt  /  Planning  &  Prep  -  GDDS 
Situation  Understanding  -  Austin  Info  Systems 
Network  Management  -  Northrop  Grumman 
Integrated  Computer  System  -  General  Dynamics 
Unattended  Ground  Sensors  -  Textron 
Ground  Sensor  Integration  -  Raytheon 
Air  Sensor  Integration  -  Northrop  Grumman 
Ground  Comm  &  Air  Comm  -  BAE  Systems 


Unmanned  Air  Vehicles 


Class  III 


iss  II 
Class  I 


Class  IV 

Northrop 

Grumman 


ned  Qround  Vehicles 


Armed 

Robotic  Vehicle 

United  Defense 

Lockheed 

Auto.  Navigation  -  GDRS 


Small  UGV 
i  Robot 


Mule 


Logistics  &  Training 

LDSS  -  Northrop  Grumman 
PSMRS  -  Honeywell 
Training  Support 

-  Northrop  Grumman 

-  Dynamics  Research  Corp 

-  Computer  Science  Corp 

Unattended  Munitions 

NLOS  LS  (LAM) 

Intelligent  Munition  System 

Non-FCS  Elements 

T  rucks 

81  mm  Mortar 
AAFARS  HTARS 
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FCS  (BCT)  System-of-Systems  Schedule 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


SoSCOE  Update 
Network  Update 
Battle  Command 

-  WMI 

-  Mission  Planning 
-Sustainment  Tools 

Spin  Out  4 


Current  to 
Future  Force 


SoSCOE  Update 
Battle  Command 
-  DCGS-A 
UGV  Family 


*  SoSCOE  Update 

*  Battle  Command 
-Sensor  Mgmt 

*  Embedded  Training 

*  UAV  Family 


Current  to 
Future  Force 
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SoSCOE 

ICS 

Battle  Command 
-  UGS/IMS 
Control 
UGS 
IMS 

NLOS-LS 
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Challenges  unique  to  FCS 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


•  Goal  is  to  produce  a  fully  integrated  Brigade  Combat  Team 

-  LSI  contractor  performs  procurement  and  integration  of  material 

-  Build  to  evolving  doctrine  -  increased  TRADOC/User  involvement 

-  Focus  on  ‘Quality  of  Firsts’ 

•  See  First,  Understand  First,  Act  First,  Finish  Decisively 

-  Spin-Outs  field  selected  capabilities  to  the  Current  Force  BCTs  over  next  8  years 

•  FCS  is  ‘born  net-centric’ 

-  First  Army  system  to  undergo  Net-Centric  Reviews,  current  focus  of  OSD  NII/FCS 
Network  IPT  (OIPT  at  3-star  level) 

-  Evaluated  against  OSD  Net-Ready  KPP  (transitioning  from  Interop  KPP) 

-  Implements  and  extends  GIG  engineering  requirements  in  a  tactical  environment 

•  Tactical  mobile  vs  tactical  ‘short  halt’ 

-  Net-Centric  information  sharing  concepts  underly  the  FCS  ‘Distributed  System’ 
architecture 

•  Massive  amounts  of  sensor  data 

-  ~400m  bits/sec  raw  sensor  data  with  variety  of  methods  to  translate  into  usable 
information 

•  Battle  Command  on-the-move;  no  more  TOC  ‘tent  farms’ 

•  Transport  systems  procured  outside  of  FCS 

-  JTRS,  WIN-T  for  FCS  BCT 

-  SINCGARS,  EPLRS,  JNN  for  Current  Force 


FCS  is  the  largest,  most  complex  program  in  Army  history 
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OV-1 :  FCS  as  part  of  the  Joint  Fight 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


■ 


M 
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APOD 


Strategic  Airlift 


Figure  1  -  Operational  View  1  (OV-1) 


FINISH  DECISIVELY 


Transition  to  next  engagement 
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ill  \m 


FCS  Layered,  Networked 
Architecture 


FUTURE  COMBAT  SYSTEMS 


Command  BCT  system  elements  are  commonly 
developed  to  integrate  FCS  platforms  into  a  larger 
geographically  dispersed  yet  Functionally  integrated 
machine 

Battle  Command  incorporates  C2,  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR),  Embedded 
Training,  and  Sustainment 


Net  ready  information  management  element  of 
service  based  architecture 


Heterogeneous  transport  layer  enables  robustness 


Networked  battle  command,  embedded  training,  and 
supportability  developed  Technical  View  (TV-1) 
integrated  into  SoS  level  TV-1  standards  supporting 
integration 


Integrated  Architecture  Provides  Design-Phase  Flexibility  and 
Tactical  Adaptability  For  The  Networked  FCS  (BCT) 
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Supporting  Net-centric  Operations 

(OSD  Nil  Net  Centric  Checklist,  v  2.1.3,  12  May  04) 


mO  GRAM  n/3J\  /? 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


Data  Tenets 

-  Make  Data  Visible 

-  Make  Data  Understandable 

-  Make  Data  Accessible 

-  Make  Data  Trustable 

-  Make  Data  Interoperable 

-  Provide  Data  Management 

-  Be  Responsible  to  User  Needs 


I A  /  Security  Tenets 

-  Identify  Management  and  Authentication 

-  Mediate  Security  Assertions 

-  Cross  Security  Domains  Exchange 

-  Manage  Identity  and  Privileges 

-  Encryption  and  HAIPE 

-  Employment  of  Wireless  Technologies 


Service  Tenets 

-  Service-Oriented  Architecture 

-  Open  Architecture 

-  Scalability 

-  Availability 

-  Accommodate  Heterogeneity 

-  Decentralized  Operations  and  Maintenance 

-  Enterprise  Service  Management 

Transport  Tenets 

-  IPv6 

-  Packet  Switched  Infrastructure 

-  Layering,  Modularity 

-  Transport  Goal 

-  Network  Connectivity 

-  Concurrent  Transport  of  Information  Flows 

-  Differentiated  Management  of  QoS 

-  Inter  Network  Connectivity 

-  DISR 

-  Joint  Net  Centric  Capabilities 

-  Operations  and  Management  of  Transport  and 
Services 


OSD’s  Checklist  frames  the  FCS  approach 
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Transport  Radio  Systems 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


JTRS  GMR 


AMF* 


WIN-T  Point  of  Presence  (PoP) 


•  Provides  ground  vehicle  based 
terrestrial  connectivity  with  Advanced 
Waveforms  (i.e.  WNW,  SRW,  ANW*) 

•  Programmable  Waveforms  to  Support 
Multiple  Missions 


*Airborne  Network  Waveform 


•  Provides  airborne  vehicle  based 
terrestrial  connectivity  with 
Advanced  Waveforms  (i.e.  WNW, 
SRW,  ANW) 

•  AMF  will  meet  SWAP  requirements 
for  FCS  CLIV  UAV  comms  relay 
package 

•  ARC-210  Form  Factor 

*Airborne  Maritime  Fixed  Station 


•  Provide  reach,  reach  back,  interoperability 

•  Functionality 

-  NCW 

-  HNW 

-  GBS  receive 

-  Interoperability  gateway 


JTRS  HMS 


Range  Extension 


•  Provides  small  form  fit  (SFF)  for  integration  into  FCS  platforms: 
-  Dismount  Soldier 


•  Provide  communication  relay  through  the  WIN-T  PoP 
vehicles 


-  Unattended  Ground  Sensors  (UGS) 

-  Unmanned  Aerial  Vehicle  (UAV)  control 

-  Intelligent  Munition  Systems  (IMS) 

-  Small  Unmanned  Ground  Vehicle  (SUGV) 

-  Non  Line  of  Sight  -  Launch  System  (NLOS-LS) 


-I 

SFF-B  SFF-A  SFF-D 

(Soldier)  (IMS/UGS)  (UAV) 


Recon  and 

Surveillance 

Vehicle 


ARV  RSTA 


Class  III 


Provides  the  Warfighter  with  superior 
Interoperability,  Flexibility,  and  Adaptability 
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Transport  Considerations 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


•  BCT  supported  with  a  two-tier  ‘network  of  networks’ 

-  JTRS  for  lower  echelons,  multiple  waveforms  and  radio  form  factors 

-  WIN-T  for  upper  echelons  and  connectivity  to  the  GIG 

-  Spin-Outs  use  existing  and  interim/prototype  radios  (e.g.  JNN,  JTRS  surrogates) 

•  IPv6  capable,  will  have  to  support  IPv4  for  current  force/coalition  issues  after 
conversion 

•  Throughput  concerns 

-  Throughput  (including  error  rates)  for  each  waveform:  predicted  vs  delivered/actual 

-  Spectrum  allocations  for  the  waveforms  across  the  BCT  area  are  likely  to  limit 
theoretic  throughput  for  the  BCT 

•  Mobile  Ad  Hoc  Network  (MANET)  requires  tailoring  of  GIG  standards 

-  GIG  upper  protocols  like  HTTP  over  TCP  don’t  work  in  MANET  environment 

-  Throughput,  availability,  network  topology  stability/rate  of  change 

•  Voice,  Video  and  Quality  of  Service  concerns 

-  Demand  far  exceeds  supply 

-  Need  for  transport-level  QoS  and  then  application  level  network  management 

-  Integration  of  FCS  net  management  with  Joint  NetOps 


Transport  is  oversubscribed  -  must  get  right  info  to  right  user  to  right  time 
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IA  Considerations 


Pl?0  GRAM  n/iJ\  i\M\GI=  /? 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


•  Cross  Domain  Solutions  -  provide  interaction  between  classified, 
unclassified,  and  coalition  networks;  issues  include: 

-  Metadata  standards  needed  to  ensure  interoperability 

-  Data  sharing  policies  to  allow  automated  release  and  provide  for  need-to- 
share  and  need-to-know 

-  Improved  certification  procedures  for  multiple  instantiations 

•  Key  Management  Infrastructure  -  provide  automated  cryptographic 
key  distribution;  issues  include: 

-  Specific  interface  requirements  for  KMI  capabilities 

-  Size,  Weight,  and  Power  constraints  on  KMI  compliant  equipment 

-  Timely  delivery  of  national  KMI  capabilities  to  support  FCS  needs 

•  Public  Key  Infrastructure  (PKI)  -  provide  authentication  capabilities  in 
line  with  DoD  PKI  strategy;  issues  include: 

-  DoD  Standard  Token  suitable  for  battlefield  use 

-  Use  of  token  for  classified  networks 

-  PKI  protocols  and  architecture  suitable  for  mobile  ad  hoc  networks 

Distributed  tactical  systems  provide  challenges  to  current  IA  approaches 
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Core  Services  -  SOSCOE 


Pl?0  GRAM  n/iJ\  i\M\GI=  /? 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


•  System  of  Systems  Common  Operating  Environment  -  SOSCOE  -  provides 
the  Core  Service  implementation  for  FCS 

-  SOSCOE  fills  4  functions  within  FCS 

•  Integrates  info  over  tactical  networks  according  to  net-centric  tenets 

•  Extends  and  federates  FCS  with  GIG  (l.e.  NCES)  Core  Services 

•  Product  line  weapon  syste m/e m bedded  software  applications 

•  Infrastructure  for  FCS  Battle  Command/Distributed  Systems  applications 

-  SOSCOE  is  developed  by  the  LSI,  and  is  85%  COTS  (by  SLOC) 

•  Multiple  OSD  evaluations  have  focused  on  SOSCOE 

-  Implicit  starting  point  has  been  “Does  SOSCOE  duplicate  NCES?” 

-  Each  study  has  concluded  tactical  transport  considerations  limit  potential  code 
reuse  between  NCES  and  SOSCOE 

•  Limited  bandwidth,  effect  of  ad-hoc  networking,  rate-of-change  on  the  network 

•  FCS  safety-critical  &  real-time  processing  requirements 

-  Most  recent  study  emphasizes  federating  core  services  across  the  GIG 

•  Study  examined  multiple  programs  of  record  that  implement  GIG  Core 
Services 

•  IA,  Collaboration  (e.g.  chat/IM),  service  management  are  high  payoff  targets 

•  NCES  has  privileged  role  to  drive  the  core  service  Key  Interface  Profiles 
(KIPs)  for  GIG  integration 

Common  infrastructure  within  FCS  and  to  the  rest  of  the  GIG 
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FCS  Open  Architecture  Provides  Multiple  Levels  of 
Interoperability  and  potential  for  early  “spiral  out” 


17/t  M  AfAAMGfff 


BRIGADE  COMB  A  T  TEAM 

One  Team-The Army/Defense/todustoy' 


Current 

Force 

Systems 

(Open) 


Current 

battlefield 

Systems 

(Closed) 


Current  System 
Integrated  through 
FCS  Interoperability 
Services 


Service 

Protocols 

Service 

^^■Discoverv 

■mgr* 

Benefits  of  Interoperability  Approach 
-  NCOW  RM  compliant  services 

lanaged  Connector!  allow  /SOSCOE 
componentsTo  use  the  services  in  other 
systems  (NCES,  etc.) 

Common  services  Domain/COI  services 
(track  management  through  SIAP,  etc.) 


Use  of  Common  COI/C2  Services 


Rev  A 
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FCS  approach  to  Service  Oriented  Architecture 
(SOA)  -  Task  Integration  Networks  (TINs) 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


•FCS  implements  its  SOA  using  Task  Integration  Networks 

-  A  Task  Integration  Network  is  the  definition  of  a  job  (the  work  to  be  performed  by  a 
user)  as  a  network  of  tasks  that  must  be  executed  in  some  order 

-  Controls  and  Sequences  Services  (both  FCS-hosted  and  Services  provided  by  other 
parts  of  the  GIG)  via  a  ‘scripting  language’ 

-  Separates  doctrinal  considerations  from  hard-coded  software  implementations 

-  Executed  by  SOSCOE  TIN  Services 

•  In  FCS,  a  TIN  is  associated  with  a  user  role  (e.g.  Bn  Cdr,  Bde  XO) 

•  FCS  TINs  federate  with  Web-Service  based  SOA  infrastructures 


-  E.g.  NCES  UDDI/WSDL 

-  Federation  occurs  at  WIN-T  PoP  (where  we  can  best  support  transport  assumptions 


L- 

o 


5 


0 


t/> 

CO 


Mission  execution 


Ingress 


Perform  area  recon 
Conduct  recon 


D 


Make  reports 


P  -- 


Egress 


FARP  servicing 


Continual  tasks 

Monitor  route  &  OPs 

Monitor  danger 
-  Monitor  ETA 


Respond  to  battlefield  events 

Initialize 

_  Compute 

intervisibility 

Determine 

route 

planning  data 

TIN-based  SOA  implements  FCS 
Battle  Command,  Training  and  Sustainment 
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Data  &  Applications  -  COI  interfaces 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Intfusiry 


•  OSD’s  intent  is  for  Communities  of  Interest  (COIs)  to  drive  application 
integration 

-  COIs  are  to  be  collaborations  of  users,  material  developers,  Service/Agency 
oversight,  etc. 

-  COIs  specify  data  items/data  relationships  of  interest  to  that  COI 

-  COIs  should  also  specify  services  of  interest  to  a  COI 

•  E.g.  Blue  Force  Tracker  COI  may  specify  a  “deconflict-this-target”  service 
interface  that  each  program  would  implement 

•  A  Battle  Command  application  would  invoke  the  common  interface  across  the 
systems  on  the  GIG,  asking  each  “OK  to  shoot  here?” 

•  COIs  will  define  a  ‘data  and  service  bus’  that  each  program  of  record  will 
implement 

•  However,  COI  governance  is  still  a  concern 

-  What  exactly  does  a  COI  produce?  Schemas,  Services,  CONOPS? 

-  When  and  how  will  COIs  produce  their  products?  (including  schedule  &  funding) 

-  How  is  a  COI  ‘good  idea’  translated  into  formal  program  requirements? 

-  How  will  overlaps/conflicts  between  COIs  be  adjudicated? 

-  How  will  COI  guidance  be  maintained/evolve  over  time  (including  changes  to 
existing  systems)? 

-  How  many  COIs  are  there,  and  how  do  you  find  the  right  COI? 


COIs  are  critical  to  the  success  of  the  DoD  Data  Strategy 
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COI  based  Net-Centric  Service  Interfaces 
(our  end-state  visjon) 


mO  GFZAIV7  IVIJ\  tSiAGi=z  /? 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


Net-Centric 

Infrastructure 


Known 
Set  Sen/ice 
Interfaces 


Systems/programs/COIs  are  for  example  only 


FEDERATED  NET-CENTRIC  INFRASTRUCTURE  (NCES/SOSCOE/piB  10.2) 


Programs  of 

Record  NECC - 

GCSS-A  -A-- 

TBMCS  — 1- 

AFATDS  - - 

FCS  - 

New  System  (e.g.  ISR) 
(Unanticipated) 


Each  System  Implements  Service  Interfaces 
Coordinated  Through  COI  Activities 
KNOWN  INTERFACES  -  UNANTICIPATED  (but  validated)  USERS 
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Net  Ready  KPP  approach 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


•NR  KPP  has  four  components 

-  Required  DoDAF  Architecture  Products 

-IA 

-KIPs 

-  NCOW-RM 

•FCS  is  actively  working  with  J6,  OSD  and  DISA  (e.g  core  service 
standards)  on  selecting  KIP  standards  and  validation  approaches 

-  KIP  definitions  are  in  flux;  J6  has  a  schedule  for  the  first  set  of  KIPs 

-  Test  and  evaluation  of  KIP  conformance  is  still  evolving 

-  FCS  is  working  with  JITC,  CTSF  and  ATC  on  who  and  how  FCS  established 
KIP  conformance 

•NCOW-RM  is  a  useful  tool  for  organizing  net-centric  information 

-  ‘Conformance’  being  worked  -  current  expectation  is  a  mapping  of 
program/system  components  to  each  leaf  node 

-  Some  NCOW-RM  nodes  are  not  just  ‘material  solutions’  -  e.g.  “Manage  the 
network”  (where  there  is  both  a  material  component  and  a  DOTL  PF 
component  for  who/when/where  the  tools  get  used) 

NR-KPP  consolidates  activities  FCS  was  already  executing 
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FCS  Lessons  Learned 


BRIGADE  COMBAT  TEAM 

One  Team-The  Army/Defense/Industry 


•Net  Centricity  has  been  a  major  impact  on  FCS 

-  Net-Centricity  is  really  a  property  of  the  System-of-Systems,  not  of  any 
individual  platform 

-  Approaches  within  DoD  (e.g.  Web  Service  SOA)  need  to  be  tailored  to  work 
in  a  tactical  transport  and  platform  environments 

-  FCS  has  had  to  defend  its  approach  across  multiple  OSD  reviews 

•  Net-Centric  Review/Net-Centric  Checklist 

•  Program  Networks  IPT  structure 

•  PA&E/QDR/PDM-III  challenges  on  potential  duplication  of  core  services 

•  All  of  these  take  PM  and  contractor  time  to  support 

•Net-Centric  Review  focuses  on  4  areas 

-  Transport,  I  A,  Core  Services,  Data  &  Apps 

-  These  are  not  easily  supported  from  DoDAF  products  or  other  standard 
formats 

•COI  impacts  are  a  big  unknown 

-  Governance,  schedule,  content  of  COI  recommendations  not  clear 

-  COIs  are  key  to  the  Data  Strategy,  so  we  have  to  make  them  successful 


“Be  Joint  or  Die”  -  Net-Centricity  is  a  PM  survival  imperative 
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From  Science  to  Solutions™ 
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Motivation 


■  Changes  in  weapons  systems 

-  Increased  ranges 

-  Complexity  of  environment 

■  Horizontal  convergence 

-  Live  -  Virtual  -  Constructive 

■  Vertical  convergence 

-  Analysis 

-  Testing 

-  Training 

-  Mission  rehearsal 

-  Operations 
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Component  Systems 

■  OneSAF  Objective  System  (OOS) 

-  Provides  context,  environment  and  synthetic  convoy 

■  Common  Architecture  Desktop/Embedded  Trainer  (CADET) 

-  Provides  a  virtual  simulator  with  the  embedded  high  fidelity 
vehicle  simulation  using  MATLAB/C++  model 

■  Talon  Robot 

-  A  live  robot  that  is  capable  of  interacting  in  the  synthetic 
environment 

■  Expedition  Dismounted  Infantry  (Dl)  representation 

-  Provides  a  dismounted  infantry  immersive  environment 

■  Test  and  Training  Enabling  Architecture  (TENA) 

-  Functions  as  middleware  for  live  testing 

■  Unmanned  Systems  Test  Bed  (USTB) 

-  Emulates  an  unmanned  aerial  vehicle  (UAV) 

■  Modular  Analysis  Test  Support  System  (MAnTSS) 

-  Collects  and  analyzes  testing  data 
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Notional  System  Architecture 


■  Provides 
|  Context 


Controls  Embedded  in 


TENA 

DIS 
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CADET  Crew  Station 


External 

Interface 


00S 

(modified) 


High  fidelity 
vehicle  model 
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Talon  UGV 


Live  system 

-  Controlled  by  OOS 

•  Replaced  lower  level  synthetic  elements  with  actual 
drivers 

-  Teleoperated  by  joystick 

Turret 

-  Remote  camera 

-  Blank  firing  M-1 6 

Wireless  Networked 


Physical 

Entity 


Control 

Mechanisms 

Organizational 

Behaviors 


Simulated 

Entity 


-  802.11 


Individual 
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From  Science  to  Solutions™ 


■ 


TENA 


■  Lightweight  communication  designed  to  connect  live 
systems 

■  Domain  specific  optimization  over  traditional  interoperability 
protocols 

■  Emerging  standard  for  range  systems 


■  Time  Space  Positioning  Information  (TSPI)  Object  Model 


+time 


♦position 


-3H  <<TENA::LocalClass» 

Time 


r 


«TENA::LocalClass>> 

TSPI 


m 


♦velocity 


>|<<TENA::LocalClass» 

Position 


-^1  <<TENA::LocalClass» 

Velocity 


♦acceleration 


<<TENA::LocalClass» 

Acceleration 


♦orientation 


♦angularVelocity 


7*|«TENA::LocalClass»  IO 

Orientation 


-H<<TENA::LocalClass» 

AngularVelocity 


►angularAcceleration 


>|  <<TENA::LocalClass» 

Angular  Acceleration 


«TENA::Class>> 

SpatialReferenceFrame 

♦srf :  SRFenum 
♦orm :  ORMenuni 


<<TENA::Class» 

All  SRF  Subclasses 
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Notional  Scenario 


■  Location 

-  Kauai  Pacific  Missile  Range  Facility  (PMRF) 

■  Entities 

-  UAV  (synthetic  from  USTB) 

-  Robotic  entity  (live  on  blocks) 

-  Stryker  variant  (controlled  by  crew  station) 

-  Trucks  /  Targets  (synthetic  from  OOS) 

-  Human  (synthetic  from  Expedition  Dl) 

■  Actions 

-  The  UAV  sees  a  small  (3-4)  convoy  of  trucks 

-  The  Stryker  moves  to  and  engages  the  trucks 

-  The  human  and  robot  inspect  the  damage  to  the  trucks 
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Scenario  Participants 


■  Vigilante  UAV 

-  Driven  from  the  USTP 

-  In  reconnaissance  mode 

■  Four  trucks  (targets) 

-  Generated  from  OOS 

-  Convoy  driving  down  road 

■  Stryker  Vehicle 

-  Driven  by  the  crew  station 

-  Hybrid  electric  drive  model  controls  dynamics 

-  Attacks  convoy 

■  UGV 

-  Physical  device  on  blocks 

-  Tasked  by  OOS/Crew  Station 

-  Inspects  convoy  after  attack  by  Stryker 


From  Science  to  Solutions™ 


Scenario 


i 
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Interoperability  Issues 


■  TENA 

-  TSPI  object  model 

-  Reference  version  was  used 

■  HLA 

-  TBD FOM 

-  Version:  RTI-1.3,  Matrex  version  4.2 

■  Terrain  Data 

-  Kauai  Pacific  Missile  Range  Facility  (PMRF)  High  Res 

-  The  digital  raster  graphics  is  at: 
http://www.hinhp.org/website/hawaii/kauai/data/drq.zip. 

-  Shape  files  and  some  others  are  at: 
http://www.hinhp.org/website/hawaii/kauai/data.html. 


From  Science  to  Solutions™ 
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Results 


■  Success! 

-  By  the  time  the  show  opened  everything  worked 

-  And  then  extended  on  the  show  floor 

■  Terrain  registration 

-  Common  source 

-  Control  over  generation  process 

■  OOS  Modular  Communication  Interface  modification 

-  Modified  HLA  interface  with  TENA 

■  High  Fidelity  Engine/Suspension  Model 

-  Wrapped  in  OOS  component  model 

-  Proxy  implementation  to  remote  computer 

■  Transparent  interaction  among  elements 


From  Science  to  Solutions™ 
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Parts  Management 
Reengineering 


9th  Annual  Systems  Engineering 
Conference 

6  October  2006 

Michael  J.  Goy,  DSPO 


Outline 

■  History 

■  Definitions/Descriptions 


■  PM  as  an  Engineering  Discipline 

■  PM  as  a  Requirement 

■  PM  in  support  of  the  Warfighter 

■  Reengineering 

■  Challenges 

■  Findings 

■  Conclusions 

■  Recommendations 

■  Where  do  we  go  from  here? 


History 

■  1977:  MIL-STD-965,  Parts  Control  Program 

■  1983:  SECDEF  Weinberger  Spare  Parts  Acq  Memo 

■  1984:  DESPECDEF  Taft  DoD  Parts  Control  Memo 

■  1994:  SECDEF  Perry  Acquisition  Reform  Memo 

■  1996:  MILOHDBK-965,  Parts  Management  Program 

■  2000:  MIL-HDBK-512,  Parts  Management 

■  2004:  Reengineer  Processes 


What  is  Parts  Management? 


■  A  multi-disciplined  process  designed  to 
improve  system  supportability 

■  Reduce  Life  Cycle  Costs 

■  Improve  Reliability 

■  Improve  Readiness 

■  Improve  Interoperability 

■  Control  growth  of  Logistics  Footprint 

■  Mitigate  DMSMS  Issues 


What  is  Parts  Management? 

■  Collaboration  between  industry  and 
Government 

■  Selection  of  parts  in  the  design  phase 

■  Analyzing  reliability,  availability,  and 
quality 

■  Screening  for  commonality 

■  Reducing  the  number  of  unique  parts 


Why  Parts  Management? 


■  Selecting  the  right  parts  drives 
downstream  outcomes 

■  Cost  reduction 

■  Interoperability 

■  Readiness 

■  Current  parts  management  practices  are 
not  adequate 

■  PMRWG  recommendations  address  these 
issues 


Making  Parts  Management 
A  “Requirement” 

■  Requirement  will  add  needed  discipline 

■  Requirement  will  address  Parts  Selection 
and  DMSMS 

■  Requirement  will  address  Parts 
Management  in  Program  Reviews 

■  Requirement  will  NOT  be  a  return  to  past 


“prescriptive”  practices 


Parts  Management  in  Support 
of  the  Warfighter 


Parts  Management  will  ensure  optimum 
parts  are  used  in  design  and  provide: 

■  Better  reliability 

■  Better  availability 

■  Better  logistics  support 

■  Better  cost  savings 

■  Better  metrics  on  readiness  data  and  ROI 


The  Reengineering  Process 

■  Included  DoD  and  Industry 

■  Researched  Best  Practices 

■  Analyzed  and  Explored  Alternatives 

■  Examined  Parallel  Efforts  (PBL,  SE,  CSI) 

■  Developed  Findings,  Conclusions, 
Recommendations 


Findings 


■  Footprint  is  growing 

■  Acquisition  environment  lacks  adequate 
emphasis  on  Parts  Management 

■  Systems  Engineering  lacks  emphasis  on 
Parts  Management 


Conclusions 

■  Parts  Management: 

■  needs  to  be  a  requirement 

■  needs  to  take  a  total  system  approach 

■decision-makers  need  better  tools 

■can  be  fully  accomplished  in  a 
performance-based  environment 


PMRWG  Recommendations 

■  Restore  Parts  Management  as  an 
engineering  discipline 

■  Make  Parts  Management  a  contractual 
requirement 

■  Develop  Parts  Management  Tools 

■  Build  key  partnerships  and  relationships 


Where  do  we  go  from  here? 


I 


■  Aggressive  implementation  through 
phases: 

□  Systems  Engineering 

□  Acquisition  Policy 

□  Training 

□  Industry  Participation 

□  Advocacy 


Any  Questions? 


AN  EFFECTIVE  UML  PROCESS 


Presented  at  the 

NDIA  System  Engineering  Conference 
24  October  2006 
By 

Jeffrey  O.  Grady 

President  JOG  System  Engineering 
6015  Charae  Street 
San  Diego,  CA  92122 
jgrady@ucsd.edu 
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Who  Is  Jeff  Grady? 


CURRENT  POSITION 

President,  JOG  System  Engineering,  Inc. 

System  Engineering  Assessment,  Consulting,  and  Education  Firm 

PRIOR  EXPERIENCE 

1954- 1964  U.S.  Marines 

1964  - 1965  General  Precision,  Librascope  Div 

Customer  Training  Instructor,  SUBROC  and  ASROC  ASW  Systems 

1965  - 1982  Teledyne  Ryan  Aeronautical 

Field  Engineer,  AQM-34  Series  Special  Purpose  Aircraft 
Project  Engineer,  System  Engineer,  Unmanned  Aircraft  Systems 
1982  - 1984  General  Dynamics  Convair  Division 

System  Engineer,  Cruise  Missile,  Advanced  Cruise  Missile 
1984  - 1993  General  Dynamics  Space  Systems  Division 
Functional  Engineering  Manager,  Systems  Development 

FORMAL  EDUCATION 

BA  Math,  SDSU 

MS  Systems  Management,  USC 
INCOSE  First  Elected  Secretary,  Founder,  Fellow 

AUTHOR  System  Requirements  Analysis  (1993,  2006),  System  Integration,  System 
Validation  and  Verification,  System  Engineering  Planning  and 
Enterprise  Identity,  System  Engineering  Deployment 
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There’s  a  Problem  Out  There 


There  are  problems  in  the  way  UML  is  often  applied 

The  application  of  this  powerful  model  should: 

-  Follow  Sullivan’s  encouragement  for  unprecedented  developments 

-  Identify  SW  entities  and  requirements  from  the  top-down  so  as  to 
coordinate  better  with  the  system  engineering  and  HW  work 

-  Employ  a  common  product  entity  structure  with  the  system  and  HW 
development 

-  Provide  for  hierarchical  traceability  across  the  HW-SW  gap 

Available  DIDs  do  not  clearly  coordinate  with  application  of 
UML 

Models  not  always  saved  or  configuration  managed 
Traceability  problems  at  the  HW-SW  gap 
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Agenda 


•  Fundamentals  of  structured  analysis 

•  UML  fundamentals  using  a  top-down  approach 

•  Hardware-software  requirements  traceability 

•  The  future  of  requirements  analysis  modeling 
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Structured  View  of  a  Problem  Space 
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Structured  Analysis  Methods 
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How  Should  I  Enter  Problem  Space? 


It  is  not  clear  how  one  can  have  a  system  that  at 
the  highest  level  is  software 

Software  must  operate  inside  of  some  kind  of 
hardware  entity 

Therefore,  I  would  elect  to  use  a  system  modeling 
approach  for  initial  problem  space  entry  where 
the  need  is  the  ultimate  function 

Traditional  structured  analysis  is  such  an 
approach 


A-7 


©JOG  System  Engineering,  Inc. 


Traditional  Structured  Analysis  Model 

Overview 
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Common 

Product  Entity  Structure 


•  Use  a  common  structure  that  includes  hardware  and  software. 

•  Apply  a  top-down  analysis  for  both  that  contributes  to  the 
identification  of  entities  in  the  common  product  entity  structure. 


A-9 


©JOG  System  Engineering,  Inc. 


A  Preferred  Modeling  Order 

Early  object  oriented  analysis  encouraged  this  pattern. 


We  will  follow  Sullivan’s  encouragement  in  this  case  -  form  follows  function  - 
because  it  coordinates  with  traditional  structured  analysis. 


UML  can  actually  support  either  direction  like  any  good  modeling  approach. 


Note:  A  classifier  is  a  general  term  for  a  software  product  entity 
represented  by  a  node,  component,  or  class  in  UML  but  by  a  block  on 
the  product  entity  diagram  like  any  other  entity. 
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The  Software  Development  Process 

Using  UML 


Identify  an  initial  product  entity  that  will  be  developed  as 
computer  software  using  traditional  structured  analysis. 


Dynamically  analyze  the  entity  using  UML. 

-  Use  cases 
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And  the  process  continues  to  expand  and  move  deeper 
translating  problem  space  into  solution  space. 

At  the  bottom  are  classes  about  which  code  can  be  written  based 
on  requirements  derived  from  the  dynamic  modeling  work. 
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Surprise! 


Note  that  the  SW  work  pattern 
encouraged  exactly  parallels  that 
employed  in  TSA. 
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The  Diagrams  of  UML  2 

For  modeling  dynamic  aspects  of  the  system 
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-  Timing  diagram 

-  Interaction  overview  diagram  (2) 

For  modelinq  static  aspects  of  the  system 

- W - " - 1 - 


Covered  in  this 
discussion 


Deployment 


Object  and  class  diagrams 
Component  diagram 


diagram 


Classif  ers 


-  Composite  structure  diagram  (2) 

-  Package  diagram  (2) 


(2)  =  added  in  UML  2.0 
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Context  Diagram 


Borrowed  from  Modern  Structured  Analysis  to 
provide  an  organized  approach  to  use  case 
identification. 


entity  the  specification  is  being 
written  for. 


®The  terminators  reflect  necessary 
external  influences  between  the 


system  and  its  environment. 
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Hierarchical  Modeling  Relationships 


-  The  Supporting  Dynamic  Modeling  Artifacts 


NOTE:  Only  a 
single  analytical 
string  has  been 
expanded  here. 
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Use  Case  Fundamentals 


Actor  Name 

A  use  case  is  a  more  expressive  form  of  the 
context  diagram  used  in  modern  structured 
analysis. 

A  use  case  bubble  represents  some  aspect 
of  the  system  being  developed. 

An  actor  represents  some  external  agent 
gaining  benefit  from  the  system. 
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Use  Case  Relationships 


Actors  derive 
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Use  Case  Relationships 


Extend 

-  Pushes  common  behavior  into  other  use  cases  that  extent  a 
base  use  case 

Include 

-  Pulls  common  behavior  from  other  use  cases  that  a  base  use 
case  includes 

Generalization 

-  A  child  use  case  inherits  behavior  and  meaning  of  the  base 
use  case 

-  The  child  use  case  may  add  or  override  the  behavior  of  the 
base  use  case 
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Use  Case  UX1 1 


Actors  derive 
benefits  from 
the  system. 


The  word  extend 
is  used  here  in  a 
generic  way 
here  to  embrace 
extend,  include, 
and 

generalization 

relationships. 
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Possible  Multiple  Scenarios 


The  word  extend  is 
used  in  a  generic 
way  here  to 
embrace  extend, 
include,  and 
generalization 
relationships. 


Textual  scenario  descriptions  (use  case  specifications) 
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Scenario 


*  A  sequence  of  actions  that  illustrates  behavior. 

*  A  scenario  may  be  used  to  illustrate  an 
interaction  or  execution  of  a  use  case  instance. 

*  Text  description  that  can  be  captured  in 
paragraph  3.1.2.h.i.j.k  of  the  classifier 
specification. 

Document 
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The  Other  Dynamic  Models 


Sequence  Diagram  UX11321 


ACTOR  CLASSIFIER  CLASSIFIER 

1  1  2 


Communication  Diagram  UX11322 
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Sequence  Diagram  UX11321 

Emphasizes  the  time  ordering  of  messages 


It  is  understood  that  the  classifiers  are  performing 
operations,  possibly  modeled  in  activity  or  state 
diagrams,  relative  to  the  message  content. 
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Messages  Between  Lifelines 


A  message  is  the  specification  of  a  communication  among 
classifiers  on  a  class  or  object  diagram  or  between  the 
classifier  represented  by  life  lines  on  the  sequence  diagram 
or  blocks  of  a  communication  diagram. 

When  a  message  is  passed  from  one  classifier  to  another 
some  action  usually  results  on  its  receipt. 

The  action  may  result  in  a  change  of  state  in  the  classifier 
on  the  arrow  head. 

Describe  the  related  requirements  in  terms  related  to  the 
target  classifier. 
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Sequence  Diagram  Message  Types 


Call 

-  Invokes  an  operation  on  an  object  represented  by  the  lifeline 

-  An  object  can  send  a  call  to  itself  resulting  in  a  local 
invocation 

Return 

-  Returns  a  value  to  the  caller 

Send 

-  Sends  a  signal  to  an  object 

Create 

-  Creates  an  object 

Destroy 

-  Destroys  an  object 
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The  UML  Static  Entities 


System/Subsystem 

-  The  highest  level  software  entity.  There  can  be  many  of  these  entities  in  a  real 
system  composed  of  hardware  and  distributed  software.  A  collection  of 
subsystems  composed  of  nodes  or  simply  nodes. 

Node 

-  Appears  on  a  deployment  diagram  that  exists  at  run  time  and  is  a  computational 
resource,  generally  having  at  least  some  memory  and  often  processing 
capability.  A  collection  of  components. 

Component 

-  A  modular  part  of  the  system  consisting  of  other  components  or  directly  of 
classes. 

Class 

-  A  description  of  a  set  of  objects  that  share  the  same  attributes,  operations, 
relationships,  and  semantics. 

Object 

-  An  instance  of  a  class. 
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>-Fo 


Deployment  and  Component  Diagrams 


Customer 
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UML  Structural  Artifacts  in  a  Product 

Entity  Structure 


Top-Down 

Development 

Encouraged 


Dynamic 

Analysis 
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A  Flexible  Dynamic  Modeling  Overview 


-Cycle  to  Lower  Tiers- 


CD  Context  Diagram 

|  Terminator  1  |  Terminator  2  | 


CD 


(5A) 

Sequence  Diagram 

!=□  [=□  □  □ 


Use  Case)  ToP  Level 
/  Use  Case  for 

Each  Terminator 


Possible  Extended  and/or 
Included  Use  Cases 


Communication 
(5B)  Diagram 


Activity 
Diagram  for 
Each  Scenario 


Dynamic  Analysis 


C\)  State 
Diagram 


Activity  Diagram 
With  Swimlanes 


Product  Entity 
Structure 

®  a 


K 

K 

K 

K 


Requirements 


Specifications 
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Organizing  the  Dynamic  Modeling 


Use  a  context  diagram  to  organize  the  use  cases. 

Recognize  a  family  of  use  cases  if  necessary. 

If  use  cases  complex,  recognize  two  or  more  scenarios  for  each 
lowest  tier  use  case. 

For  each  scenario,  build  a  sequence  or  activity  diagram  and  in  the 
process  identify  next  lower  tier  classifiers  and  messages  between 
the  actors  and  lower  tier  classifiers. 

Apply  communication,  activity  or  sequence,  and  state  diagrams  as 
needed. 

Derive  requirements  from  dynamic  modeling  artifacts  and 
relationships. 
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Traceability  Forms 


*  Vertical  requirements  traceability 

-  Hierarchical  or  parent-child 

-  Requirements  source  traceability 

-  Requirements  rationale  traceability 

*  Longitudinal  traceability 

-  Requirements  to  synthesis  and  verification 


•  Lateral  traceability 

-  Traceability  to  method 

*  Applicable  document 

-  Internal  integrity 


LONGITUDINAL 

TRACEABILITY 
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System  Product  Entity  Structure 


1  1  Software  Entities 

“  ~  Requirements  Traceability 
Concerns 


Downward  Traceability  Situation 
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The  System  Product  Entity  Structure 


1  Hardware  entity 
I  Software  entity 


This  is  the  kind 
of  relationship 
of  interest 


Level  at  which  a 
subordinate  software 
entity  is  identified 
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um  a 

Traceability  Across  the  Gap 
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Function  FT  within  TSA  application 
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D  D8U776 
many  other 


requirements  from  multiple  functions 
Context  diagram  terminator  UX21 
Use  case  UX211 
Extended  Use  Case  UX2111 
Scenario  UX21111 
Sequence  diagram  UX211111 
Software  requirement  RID  894RT5 
derived  from  the  sequence  diagram 
RID  894RT5  traceable  to  one  of  the 
requirements  allocated  to  AX2  using 
TSA. 


SOFTWARE 
ENTITY 
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THROUGH  TSA 
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AX1 


CONTINUED 
TRADITIONAL 
STRUCTURED 
ANALYSIS  FOR 
AX1  AND  AX3 

THESE  ARE  THE  MODEL 
ID  (MID)  THAT 
REQUIREMENTS 
DERIVED  FROM  THESE 
MODELING  ARTIFACTS 
ARE  MAPPED  TO  IN  THE 
RAS  COMPLETE  (IN  THE 
BIG  DUMB  DATABASE) 
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Sequence  Diagram  UX211111 
Communication  Diagram  UX211112 
Activity  Diagram  UX211113 
State  Diagram  UX211114 


PRODUCT 

ENTITY 

AX3 


Entity  AX2 

Context  Diagram  Terminator  UX21 
Use  Case  UX211 
Extended  Use  Case  UX2111 
Scenario  UX21111 
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Traceability  Evaluation  Matrix 


CONTEXT  DIAGRAM 
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Requirements  Derived 
From  TSA  Modeling 


Traceability  Evaluation  Matrix 
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POSSIBLE  RID 
EXAMPLE 


R%1 

RU7Z7H 

R%2 
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R%3 
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R%4 

RJ8E6G 
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RJYT6T 

R%6 

RHGT5T 

R%7 

RID87W 

R%8 

RBJ8S7 

R%9 

RL34DF 

R%1 0 

R456HD 

Alternatively,  one  could  rely  upon  experienced  inspection  without  the 

organizing  influence  of  the  matrix. 
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Save  the  UML  Models  Too! 
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Combined  SDD 


System 

Definition 

Document 


Alternative 

Capture 


Document  Body 

Appendix  A,  System  Functionality 
Appendix  B,  System  Environment 
Appendix  C,  Product  Entity 
Appendix  D,  Interface 

Appendix  E,  Specialty  Engineering 
Appendix  F,  Process 

Appendix  G,  Context  Diagrams 
Appendix  H,  Use  Case  Diagrams 
Appendix  I,  Scenario  Text 


TSA  Application 
Graphics  Application 
UML  Application 
General  Database  (Such  as  DOORS) 


Appendix  J,  Sequence  Diagrams 

Appendix  K,  Communication  Diagrams 
Appendix  L,  Activity  Diagrams 
Appendix  M,  State  Diagrams 
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Tools  Integration 
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An  Interim  Integrated  Tool  Set 
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UML  and  Functional  Analysis 


Unified  Modeling  Language  (UML) 
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Modeling  Changes  In  the  Near  Term 
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Agenda 


Recap  implications  of  complex  product  development 

-  Geographical  distribution;  explosive  growth  in  technical  data  and 
management  challenges 

Analyze  technical  data  management  infrastructure  requirements 

-  Program  planning 

-  Data  production  and  use 

-  Program  performance  assessment 

Outline  an  approach  to  managing  work  products 

-  Based  on  using  work  product  descriptions  to  specify  templates  with 
verification  conditions 

-  Which  enables  better  evidence-based  program  performance  assessment 

Challenges 

-  Finding/developing  usable  technical  standards 

-  Integration  across  distributed  legacy  data  repositories 

-  Cultural  change 

Where  we  go  from  here 

-  Adopting/developing  technical  data  standards 


Complex  Product  Development  Environments 

Are  Globally  Distributed 


PAX  Server 


JSF  Data  User 
Community 


Network  To  PAX 
♦  . 


Archived  Test  Data 

*♦1 

Network  To  EAFB 
>* 


Fort  Worth 
Data  Center 


EAFB  Server 


EDW 


IPTs 


Etc.,  Etc. 


Terabytes  of  Data  Must  Be  Accessed  World  Wide 
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Explosive  Growth  In  Digital  Data  Volume 

...  overwhelms  the  ability  of  humans  to  know  where  data  is ,  how 
data  is  organized ,  and  what  it  means 


Data 

models  & 
schemas  < 
used  to 
organize 
data 


"N 


> 


J 


Gigabytes  of 
metadata 


Terabytes  of 
data 


The  volume  of  data  on  programs  is  now  in  the  terabytes.  Individuals 
are  generally  forced  to  rely  on  specifics  of  how  each  repository  is 
organized ,  which  varies  enormously. 


Work  Products  (Technical  Data)  Provide 

...the  evidence  of  program  performance 


Work  Product 
Planning 


•  Descriptions  of  work 
products  to  be  produced 

•  Tech  Plans 

•  Templates 

•  Processes 

•  Schedules 


Release  looks  at 
individual  work  products 


Work  Product 
Production  & 
Release 


Assessment  looks  at  the 
collection  of  work 
products  to  determine 
data  consistency, 
design  maturity,  ... 


•  Work  Products 
validated  per  plans  and 
schedules 


Program 

Performance 

Assessment 


•  Evaluation  for 
completeness  and  for 
program  maturity 
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Managing  Technical  Data  Requires 


Descriptions  of  work  products  for 

-  Identification  and  validation 

-  Locating  and  understanding 

Specification  for  each  work  product  that 

-  Defines  format  and  content 

-  Facilitates  quality  and  maturity  assessment 

Specifications  that  apply  to  collections  of  work 
products  for  assessment  at  decision  events 

-  Completeness 

-  Consistency 

-  Program  technical  maturity 

-  Impact  of  changes 


A  Work  Product  Is  ... 


•  A  document,  report,  analysis,  test 
result,  plan,  etc.  that  is  generated 
by  the  product  development 
process 

9  Specified  with  a  well-defined 
purpose  in  the  development 
lifecycle  with  identified  producers 
and  consumers 

9  With  unique  identification, 

release  and  configuration  control 

9  A  work  product  may  contain 
components  including  models 
and  drawings 
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WORK  PRODUCT 
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... work  products  are  the  primary  evidence  of  program  execution 
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A  Work  Product  Template  ... 


•  Describes  the  structure  and 
content  of  a  work  product 

•  Specifies  valid  choices  for  work 
product  attributes  such  as  IPT 
names,  product  versions,  etc. 

•  Specifies  verifiable  requirements 
for  work  products 

•  A  template  represents  the  starting 
point  for  author  of  a  work  product 

•  A  template  supplies  pre-filled  in 
content  and  relationships  where 
possible 

•  A  template  can  be  used  to  generate 
an  interactive  form  to  construct  the 
work  product  with  choice  lists  for 
attribute  values  (automated 
validation) 
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Problem:  How  To  Construct  Templates  That 


Cover  all  planned  work  products 

Provide  maximum  uniformity  and  commonality  across 
all  work  products 

Provides  a  specification  of  identity  up  to  versioning 

-  Any  work  products  developed  from  the  same  template  are 
versions  of  each  other 

Are  machine-interpretable  with  rules  for  machine 
processing 

-  used  to  provide  validation  for  structure  and  content 
requirements  on  work  products 


Work  Products  Have  Structure 


...  that  defines  commonality  and  uniformity  that  need  to  be 
specified  by  templates 


•  Specialization 

-  Level  of  detail,  e.g.,  functional, 
system,  physical  specifications 

-  Different  tiers  in  the  product 
structure,  e.g.,  avionics,  radar 

•  Specific  components  and 
interrelationships 

-  E.g.,  a  functional  specification 
contains  or  references  many 
specific  component  work 
products 


Verification 

Condition 
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Work  Products  Participate  In  Relationships 

...  that  are  used  for  design  traceability  and  product  maturity 
assessment  that  need  to  be  defined  by  templates 
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Our  Approach  Uses  a  Work  Product  Description 


... to  specialize  a  generic  template  to  produce  a  template  for  a  work 
product  with  a  given  description  Description 


User  Authoring  Scenario - 

1.  User  presented  with  a  valid  list  of 
versions 

2.  Choice  of  version  used  to  determine 
possible  components/subsystems 

3.  Choice  of  version,  product  component 
determines  admissible  work  product 
categories 

4.  User  determines  category  from  filtered 
list  of  categories,  which  determines  valid 
topics 

5.  User  choice  then  builds  content  and 

organization  for  template - 


Product  Version:  F-35  Carrier  Version 
Product  Structure:  Avionics 
Work  product  category:  Specification 
Topic:  Functional 


_ /Template 

TITLE:  CV  Avionics  Functional 
Specification 

Allocation  of  Air  Vehicle 

Functional  Block  Diagram 

Functions  with  performance 
conditions  and  verification 
conditions 

External  Interfaces 


...  the  work  product  category  with  its  topic  specialization,  as  well  as 
the  product  version  and  product  structure  component  all  contribute 
to  defining  a  specific  template 
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Template  Specialization  Depends  on  Work 
Product  Descriptive  Attributes 

...  the  attributes  below  determine  work  product  templates 

•  Product  Version 

-  The  specific  configuration  description  of  the 
product,  e.g.,  Carrier  Version 

•  Product  Component 

-  The  specific  subsystem  or  configuration  item 
of  the  product  version 

•  Work  Product  Category 

-  Specifies  structural  components  of  a 
template,  e.g.,  plans,  specifications 

•  Topic 

-  Specializes  the  category  for  a  specific  subject 
matter  domain,  e.g.,  external  loads 

...  the  work  product  category  with  its  topic  specialization ,  as  well  as 

the  product  version  and  product  structure  component  all  contribute 

to  defining  a  specific  template 


Work  Product  Categories  Can  Be  Defined 
From  Bottom  Up  or  Top  Down 


...there  are  lots  of  gray  areas  here  and  choices  to  be  made 
concerning  whether  the  categories  are  descriptive  or  prescriptive 
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The  Benefits  of  the  Description 
&  Template  Approach 

•  Planners 

-  Provides  sufficiently  detailed  descriptions  of  planned  work 
products  that  can  be  for  machine  validation 

•  Producers 

-  User  friendly  way  to  author  work  products  with  pre-filled  in 
content  and  machine  checked  work  product  validation 

•  Users 

-  Enables  location,  access,  understanding  through  descriptions 

•  Examiners 

-  Enables  completeness  checking  for  events 

-  Identification  of  supersession 

The  description  and  template  approach  can  be  used  for 
program  assessment  even  when  approach  is  not  fully 

automated 
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Determining  Descriptive  Classifications  &  Relationships 

...is  an  example  of  “ontology  engineering ”  in  the  domain  of 
technical  data  management 

Ontology  Engineering:  Defining  terms  and  relations 
among  the  terms  for  a  domain 

-  Defining  concepts  in  the  domain  (classes) 

-  Arranging  the  concepts  in  a  hierarchy  (subclass-superclass 
hierarchy) 

-  Defining  which  attributes  and  properties  (slots)  classes  can 
have  and  constraints  on  their  values 

-  Defining  relationships  between  classes 

-  Defining  individuals  and  filling  in  slot  values 


The  class  model  represents  the  general  domain  knowledge, 
the  instance  information  represents  specific  program 
_ knowledge _ 
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Challenges 


Standardizing  the  concepts 

-  Defining  classes  &  relationships 

-  Populating  classes  with 
instances  of  program  specific 
information 

Implementing  templates  to 
capture  and  validate  data 

-  Capture  metadata  separately 

-  Use  templates  for  interactive 
form  validation 

Providing  a  portal  with 
application  logic  to  manage 
data 

-  With  services  to  integrate 
distributed  data  repositories 


Users 


TDM  Portal 

•  use  metadata  to 
drive  portal 

•  location  &  browsing 

•  data  access 
•Audit  reporting 


Information 

Model 


...is  the  approach  practical?  The  answer  is  yes 
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Where  To  Go  From  Here 


•  Develop  a  full  Technical  Data  Management  (TDM) 
ontology 

-  Expand  the  scope  from  data  required  for  design  and 
development  to  data  required  for  operation  and  sustainment 

*  Use  Semantic  Web  framework  including  the  ontology 
languages  OWL 

-  TDM  and  Information  Portal  are  2  of  5  use  cases  listed  for 
requirements  development  of  the  ontology  language,  OWL 

*  Integrate  TDM  ontology  with  PLM  standards  (e.g., 
GEIA-927,  AP233,  AP239) 

•  Leverage  this  assessment  capability  to  improve  earned 
value  management 
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Questions/Comments 


What  Is  An  Ontology  (Natalya  Noy) 

•  .  ,  f,  **■  W  ^  ,  ,||  1  '  jg,:,  'ipUl  ..J  ;*  ;  jf^.  .  J1  W  v>  *  .  &GHH 

■  An  ontology  is  an  explicit  description  of  a 
domain: 

■  concepts 

■  properties  and  attributes  of  concepts 

■  constraints  on  properties  and  attributes 

■  Individuals  (often,  but  not  always) 

■  An  ontology  provides  a  shared  understanding 

■  the  common  vocabulary  and  rules  of  use 
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This  Diagram  Illustrates  the  Scope  of  the  STEP 
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inions  whieh  woBld  reduce  the  capability  of  the  airplane  or  the 
of  the  crew  to  cope  with  adverse  operating  conditions  [must  be] 
janle  <1  *  10-5  penfiantmilr.  less  tor,  sevemjinnditicws 


“In  general,  the  means  of  compliance  described-in* t-his-AG-ar-e 
not  directly  applicable  to  software^assessments  because 
it  is  not  possible  to  assess  the  number  and  kinds  of 
software  errors,  if  any,  that  remain  after  the  completion  of 
aBB/an.  development  and 


or  software  to  RTCA  DO-178B 


YSTEM  ASEE0T-S 


SOFTWARE  LIFE  CYCLE 
SOFTWARE  PLANNING  PROCESS 
SOITWARE  DEVELOPMfiNT  PROCESS 
SOFTWARE  VERIFICATION  BROOTSS- 


A/OT  TRACEABLE  TO  FAR  25. 1309 


LSS5i§iTrVMS.I 


ode  usually  does  its 

^bbr-B-r-eakdown s  typically  occur  when  the 
software  exception  code  does  not  properJy_ 
handle  abnormal  input  or  environmental 
conditions  -  or  when  an  irrterrSce  adbs  not" 

anticipated  or  desired 


s  of  Reliability  Engineering  Technology  2001, 
Newsletter  of  the  IEEE  Reliability  SoaetyfJaTTC^ffyzOO'i  "’l 


TEST  RESULTS  W/  ACCELEROMETER.  FAILURES 


ORTHOGONAL  OUTPUT 
-FROIW6  NON-ORTHOGO- 
NAL  ACCELEROMETERS 

PROGRAM  SHOULD 
TOLERATE  UP  TO||hREE 

acceIerometer 

FAlluRES 


No.  accel.  Total  tests  Tests  Failure 

failed  failed  fraction 


134,135  1,268  0.01 


12,921  0.13 


eckhardt;  caglayan  et  al.,  an  experimental  evaluation  of 

SOFTWARE  REDUNDANCY ,  TSE,  7/91 
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EXCEPTIONSARE 
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■low  level  failure  effects 
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FAILURE  MODE  AND  EFFECTS  ANALYSIS 


SYSTEM 

INDENTURE  LEVEL _ 
REFERENCE  DRAWING. 

MISSION _ 


DATE. 


SHEET. 


.OF. 


COMPILED  BY. 
APPROVED  BY. 


i  o  E  nt  iMCAnOh 

NUMBER 

iTEM/mCT-QNAl 

IDENTIFICATION 

(NOMENCLATURE} 

FUNCTION 

FAILURE  MODES 
AND  CAUSES 

MISSION  *HA$E/ 
OPERATIONAL 
MODI 

FAILURE  EFFECTS 

SEVERITY 

CLASS 

COMPENSATING 

PROVISION! 

FAILURE 

DETECTION 

METHOD 

remarks 

LOCAL 

EFFECTS 

Hit  T 
HIGHER 
LEVEL 

END 

EFFECT ! 

IDENTIFICATION  CAUSES  AND  EFFECTS 


DISPOSITION 


■^RAHtHR&MDDE  fRJNCTiOim)  E.  G|NO  OUTPUT 

-P/MLURE  CAUSE  (ENGINEERING)  c;  ll 

OXIDE  FAILURE  2.  BOND  BREAKAGE - — _ 

MISSION  PHASE,  OPERATIONAL  MODE 
EFFECTS 

-  LOCAL 

R  LEVEL 


91 


■aai 
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ERITY  CLASSIFICATION  BASED  ON  fl\l  D^li^CTS 


K: 


AN  BEWT  SEVERAL  LEVELS 


COMPENSATE  OB  PROVISIONS 

BrEDUNDANCY,  RETRY,  BAC^-UPm)Bg 

REMARKS 


HSJfg  1 


'HEEFFECT  I 


MO  GET 


Model-based  Certification  Tool 
ComputejAidedi  generation  of  FMEAW 
Evaluation  of  robustness^rovJsidS^ 
TPNs  fqrexploration-of-timing  "problems 


BlockType 

Name 

Position 

Value 

} 

Block  { 

BlockType 

Name 

Position 

Value 

} 

Block  { 

BlockType 

Name 

Ports 

Position 

ShowName 

IntegratorMethod 

External  Reset 

JnitialConditionSource  "internal" 
SampleTime 

} 

Block  { 

BlockType 


Position 

mputSameDT 

} 


Constant 

"Constantl" 
[155,496,  240,524] 

"FCS.lon.k5b" 


Constant 

"Constant2" 
[35,  411,  120,  439] 

"FCS.lon.k5c" 


Discretelntegrator 

"Discrete-Time\n  I  ntegrator" 
[1,1] 

[395,  160,  430,200] 
off 

"Forward  Euler" 

"none" 


"FCS.T_Samp" 


[2,1] 

[310,  383,  345,  552] 
off 


j~\r  \ 


inal  claw 
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l.l. 1.1 

Nz_cmd 

1.1. 1.2 

Pitch_FB 

1.1. 1.3 

Bus\nSelector 

H.1.3.f 

<Nz_g> 

1.1. 1.3.2 

<Q_dps> 

§1.i.3.3 

<AoA_deg> 

Ik  i  .1.4 

Constant 

f.  1.1.5 

Constantl 

11.16 

Constant2 

llj.7 

Discrete-Time\nlntegrator 

Product 

DETECT  Z 


O  OUTPUT 


Assertion 


-Geiwnand 


1.1. 1.2 

1.1. 1.3 

Bus  selector 

1.1. 1.3.1 

N  z  FB 

"T.7L1 .7  Product 

: 


Absent 

Jump 

c.  >  Limit 


See  1.1. 1.3 


No  output 
Hi  rate 

None  (limiter) 


No-  FB 


N^^tgTTcTl 
Hi  rate 


No  signal 
Hi  rate 


N-wait* 
Chck  rate* 


N-wait 


N-wait* 
Chck  rate' 


*  Not  in  current  model 
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Objectives 


Project  Objective : 

Evaluate  and  assess  NAVSEA  PEO  IWS  FORCEnet  Open 
Architecture  (OA)  functional  architecture  against  three  (3)  air 
defense  engagement  scenarios  to  either  validate  the  model  or 
identify  recommended  alternatives  and  /  or  improvements. 


Presentation  Objective’. 

Provide  overview  of  project  research  methodology,  including 
descriptions  of  models  and  architectural  frameworks  used  to 
bound  research,  and  present  recommended  model 

improvements. 


25  October  2006 
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Methodology 


•  Characterize  the  Problem  /  Battle  Space  -  identify  operational 
environment,  including  actual  engagement  scenarios,  to  bound  problem 
space 

•  Formulate  Design  Principles  -  consider  operational  environment  & 
available  FORCEnet  /  OA  models  and  strategies  in  formulating  principles 
for  architecting  OA  system  with  Integrated  Fire  Control  (IFC)  capabilities 

•  Develop  Conceptual  Design  -  develop  architectural  framework  for 
distributed  system  with  automated  decision  aids  for  optimally  managing 
warfare  resources  within  the  problem  space 

•  Functional  Decomposition  -  develop  models  and  methods  to  express 
automated  resource  collaboration  in  the  context  of  the  FORCEnet/OA 
architecture  domain 

•  Simulation  Modeling  —  develop  computer  simulation  model  to  assess 
proposed  functional  system  architecture  against  the  engagement  scenarios 

•  Analyze  simulation  Results  -  Assess  results  of  simulation  to 
determine  validity  of  previously  identified  FORCEnet  OA  model 


25  October  2006 
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Definitions 


FORCEnet:  operational  construct  and  architectural  framework 
for  naval  warfare  in  the  information  age,  integrating  warriors, 
sensors,  command  and  control,  platforms,  and  weapons  into  a 
networked,  distributed,  combat  force 

Open  Architecture: 

-  Technical  and  Functional  Architectural 

-  Interfaces  at  system  boundaries  must  be  described  in  detail  using 
established  commercial  or  DoD  standards 

-  Well-defined  and  easily  understandable  system  boundaries  at  various 
levels 

-  Subsystem  descriptions  and  interface  definitions  capable  of  two  or 
more  layers  of  decomposition  using  established  system  engineering 
practices 

-  Foundation  for  21st  Century  Combat  System  Designs 


25  October  2006 
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Definitions  (cont’d) 


•  Time-Critical  Targeting:  ability  to  detect,  track,  engage,  and 
assess  threats  having  extremely  limited  windows  of 
vulnerability  or  opportunity  for  detection 

•  Integrated  Fire  Control:  ability  of  a  weapon  system  to  develop 
fire  control  solutions  from  information  provided  by  one  or 
more  non-organic  sensor  sources,  conduct  engagements  based 
on  these  solutions,  and  either  provide  mid-course  guidance  or 
allow  this  guidance  to  be  provided  by  another  warfare  unit 

kfljj 


25  October  2006 
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FORCEnet  Information  Architecture 


Legend: 

Invariant  architectural  boundaries  with  network  control  system  “throttles”  to  assure  the  metrics 
. Kill  chains 

*  Networks  trace  to  communities  of  interest 
C  0  Networks 

Source:  FORCEnet  Implementation  Strategy,  National  Research  Council,  2006 

25  October  2006  WWW.NPS.EDU 
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OA  Warfare  System  Domain 


OA  Warfare  System  Domain 


1.0  Search  / 
Detect  (S  /  D) 


Sensor  Asset 


Sensor  Report 


Sensor  Track 
Report 


INTEL  Report 


Measurement 

Report 


s 

Comi 

Sen 


WW 


2.0  Data  /  Information 
Services  (DIS) 


System  Track 


Supporting  Source  Track 


Classification 


Track  Kinematics 


Attribute  Data 


Track  Repository 


NRT  INTEL  Track 


Sensor  Scheduler 


^ViVVsVleV 


3.0  Planning,  Assessment  & 
Decision  (PAD) 

Assigned 

Missions 

Tactical 

Picture 

Action 

Plans 

Capability 

Plan 

Threat 

Mission 

Assessment 

Assessment 
(Including  Identity) 

C2  Order,  Sc 

:hedule&  Event 

5.0  Mission 
Execution  (ME) 


Weapons  System 


Air  /  Surface  Missile 


Land  Attack  Missile 


Torpedo 


Gun  ■  Decoy 


RV  Assets 


Aircraft 


Boat 


Un-Manned  Vehicle 


Eng  Control  Sys 


Engineering 


Damage 


Bridge 


ww 


-ft-  u  s  u 


6.0  EXCOMM 


Communications 
Service  Action 


Network 

Message 

Schedule 

Event 

■ - ■ — - ■  ■ - ■ 

Network 

Radios 

Data  Links 

SatCom 

1 

w 


7.0  Common  Services  (CS) 

|  Display  |  Time 

\  NAV  | 

dx;dr  | 

Databases 

Environment 

8.0  Training  (TR) 


9.0  Force  Planning  /  Coordination  (FP  /  C) 


Joint  BF 
Orders 


Commanders 

Estimate 


COA 

Repository 


Force 

Integrated 

Scheduler 


|  j  Force  Network  |  Provided  Data 

|  |  Local  Network  (OACE)  Consumed  Data 

□  Candidate  OA  Common  Function/Application 
j  |  Candidate  OA  Platform-Unique  Function  /  Application 


Source:  Combat  Identification  for  Naval  Systems  in  an  Open  Architecture, Young,  2006 
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OA  Compliance  Categories 


Legacy 


Category  1 

Hardware 

Adapter 


Legacy  App. 


Legacy  H/W 
Legacy  OS, 
M/W,  etc. 
Physical  l/F 
adapter 


non-OACE 

Application 


non-OACE 

Environment 


Hardware 

Adapter 


OACE  App  U 

Source:  OACE 

25  October  2006 
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OACE  Compliant 

m 
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Category  2 


OACE 

Interface 


Legacy  OS  APIs 
&  Middleware 


Adaptation 
layer 


OACE  App 


Wrapper  layer 
makes  legacy 
APIs  portable 
OACE  M/W  for 
external 
interfaces 
Subsystem-level 
reuse 


non-OACE 

Application 


Cateaorv  3 

Cateaorv  4 

Enabling 

Enabling 

Portability 

S/W  Reuse 

Apps.  ported 
to  OACE  OS 
&  middleware 


Applications  built 
on  OACE  standards 


OACE  standards 
used  internally 
OACE  physical 
infrastructure 
Minimal  change 
to  application 
software  architecture 
Supports  common 


Uses  of  OA  common 
services  and 

applications  across  Force 
Applications  adhere  to 
OAFA  and  APIs 
Applications  use  OA 
frameworks  (e.g.  fault 
tolerance) 


function  reuse 

OACElApp  or 

OACE -based 

Comijion  Fn 

Application 

OA  services 

OACE 

OACE 

Technologies  and  Standards,  NSWC,  Dahlgren  Division,  4  Sep  2003 
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Legacy  Models  Considered 


OODA  Loop 


OBSERVE  I  ORIENT 


* 


Sensor 

Coordination  "Net" 
(e.g.,  CEC/JCTN) 


DCE  Sequence 


C2/Information  "Net" 
[e.g.,  JPN  (Non-Real  Time), 
JDN  (Near  Real  Time), 
CEC/JCTN  (Real  Time)l 


Weapons 

Coordination  "Net" 


Kinematics 

Force  Level: 

Unit/Force 

Unit/Force 

*  Position 

*  IERs 

Level: 

Level: 

*  Velocity  ID 

*  Info  Exchange 

Decision 

Engagement 

- n - 

Effectiveness  (IEE) 

Reaction  Time 

Decision 

Effectiveness 

Force  Virtual 
Sensor 


Command  and  Control 


Force  Weapons  EW 
and  Hard-Kill 


LEVEL  0 
Signal  Detection 
Feature  Extraction 


Data  Fusion  Model 


Level  1 

Level  2 

Level  3 

Track 

Situational 

Force  Threat 

Kinematics 

Awareness: 

Evaluation 

&  ID 

Context, 

(TEWA) 

CTP 

Planning, 

Estimating 

LEVEL  4 
Force  Weapons 
Assignment  (TEWA) 
Resource  Management 
Mission  Assessment 


Information  Grid 

(e.g.,  GIG) 
Force  Information 
Exchange 


Shooter 

(Engagement) 

Grid 
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Source:  A  Self-Consistent  Context  for  Unit  and  Force 
Level  Tactical  Decision-Making,  Luessen,  2003 


ii 


Plans 


•  Fundamental  IFC  System  Characteristics 

-  Dynamically  updateable  doctrine 

-  Decentralized  architecture  and  synchronized  information 

-  Doctrine  and  decision  aids 

^  •  Key  IFC  Capability  Requirements 

i  -  Shared  Situational  Awareness  (S A) 

-  Determining  best  Course  of  Action  (CO  A) 

Sal  -  Distributed  Resource  Management  (DRM) 

-  Embedded  IFC  planning 


25  October  2006 


Source:  Integrated  Fire  Control  for  Future  Aerospace  Warfare,  Young,  2004 
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Common  Identification  (ID) 


•  Objective:  accurate,  timely,  and  sustainable  characterization  and 
classification  of  tracks  to  facilitate  early  threat  and  resource  awareness,  and 
enable  optimal  weapon  engagement  planning 

•  Current  methods:  Identification  Friend  or  Foe  (IFF),  Local  sensor  data 
Intelligence,  Surveillance,  and  Reconnaissance  (ISR) 

•  Future  Capability  Requirements 

-  Centralized  geographical  database 

-  Shared  resource  picture 

-  Common  CID  deterministic  results 

-  Common  CID  functionality 

-  Support  Joint  Warfare  operating  environment 

-  Effective  CID  data  strategy 

-  Span  Warfare  areas 

-  Increased  level  of  automation 

-  Isolate  CID  processes  (OA  design) 

Source:  Combat  Identification  for  Naval  Systems  in  and  Open  Architecture,  Young,  2006 
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Future  Common  ID  Capability 


Other  Military  & 
Civilian  Units 


Other  Military  & 
Civilian  Units 

k  \  Radio 
\  \  Transmission 
Radio  \  \  Response 
Transmission  \  \(contains  ,D) 
Interrogation  \  L 


Battlespace  Objects 


Active 

Sensor 

Emissions 


Location  &  Status  of  Friendly 
Force  Units  &  Defended 
Assets  via  Comm  Links, 
Internet,  or  Phone 


Friendly  Force 
& 

Defended  Asset 
“Picture” 


Location  & 
Status  of 
Friendly 
Force  & 
Defended 
Assets 


Phenomenological 

Emissions 


On-Board  & 
Off-Board 
Sensors 


CID-Related  Sensor 
Data  &  Track  Data 


x 


CID  Fusion  Engine 

-  Associate/fuse  CID  data  from  various 
sources  per  battlespace  object 

-  Analyze  CID  data  to  make  CID 
determinations 

-  Synchronize  CID  determinations  with 
those  made  by  other  units 


Enemy 
Information 
(from  Intel 
sources) 


Predicted  Signatures, 
Threat  Templates, 

A  Priorit  Knowledge  of 
Enemy  Locations  & 
Capabilities 


Location  &  Status 
of  Friendly  Force 
Resources 


CID  Result  Synchronization 
(Discrepancy  Identification  & 
Resolution 


Location  &  Status 
of  Friendly  Force 
Resources  via 
Comm  Links 


Other  Military 
Units 


Off-Board  CID 
Fusion  Engine 
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postgraduate  Battle  Space  Information  Realms 


The  FORCEnet  OA  operational  concept  will  enable  ideal 
threat /weapon  pairings  by  maintaining  BF-wide  shared 
situational  awareness  across  the  three  information  realms 


COP  ~  Non-Real  Time  (minutes) 

-  Supports  Planning/ Force  Mgt: 

-  Deployment 
-  Movement 

Consists  of:  tactical  info,  courses  of 
action,  enemy  order  of  battle  info 

CTP  ~  Near-Real  Time  (seconds) 

-  Supports  Cueing/ Force  Mgt: 

-  Maneuver 

-  Asset  Control 

-  Weapon/ Sensor  Cueing 
-  Target  Weapon  Pairing 

-  Consists  of:  tactical  information 

FCP  ~  Real  Time  (sub-seconds) 

Supports  Fire  Control: 

-  Launch  &  In-Flight  Fire  Support 

-  Engage  on  Remote 

Consists  of:  fire  control  quality  measurements 
with  single  threat  focus 


Source:  Naval  Network-Centric  Sensor  Resource  Management,  Worth/Green,  2006 
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Engage  on  Remote 


Engage  on  Remote 
OV-1 


1 )  Remote  unit  provides  fire  control 
quality  threat  data 

2)  Firing  unit  launches  interceptor  based 
on  remote  threat  data 

3)  Remote  unit  continues  to  control 
engagement  (compute  and  provide 
interceptor  guidance,  etc.)  based  on 
remote  data 


Inbound 

Threat 


Threat 
Launcher 
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Forward  Pass 


Forward  Pass 


OV-1 


1 )  Firing  ship  launches  interceptor  based 
on  local  threat  data 

2)  Remote  unit  takes  over  engagement 
control  (compute  and  provide 
interceptor  guidance,  etc.)  based  on 
remote  data 


Remote  Unit 


25  October  2006 


Unit 


Inbound 

Threat 


Threat 

Launcher 
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Remote  Fire 


Remote  Fire 
OV-1 
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Remote  Unit 


1 )  Both  ships  maintain  fire  control  quality 
threat  data 

2)  Remote  unit  provides  launch  data  and 
firing  order  to  firing  unit 

3)  Firing  unit  launches  interceptor 

4)  Either  remote  or  firing  unit  can  control 
engagement  (compute  and  provide 
interceptor  guidance,  etc.)  as  required 


Relay  Fire 
Control 
Solution 


jr# 


□ 

REMOTE  UNIT 

□ 

FIRING  UNIT 

□ 

ALL  UNITS 

□ 

NOT  UTILIZED 

25  October  2006 
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□ 

REMOTE  UNIT 

□ 

FIRING  UNIT 

□ 

ALL  UNITS 

□ 

NOT  UTILIZED 

4.0 


Determine 

FC 

Solution 


Relay  Fire 
Control 
Solution 


6.0 

Coordinate 

Assets 
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Forward  Pass  FFBD 


7\0  8.0  9.0  10.0 


14.0 


Evaluate 

Engagement 

13.0  12.0  11.0 


15.0 


Monitor/ 

Report  All 

Data 
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□ 

REMOTE  UNIT 

□ 

FIRING  UNIT 

□ 

ALL  UNITS 

□ 

NOT  UTILIZED 

Relay  Fire 
Control 
Solution 
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Simulation  Model 


Determine 

FCS 


Fire  Weapon 


End  Threat 
Engagement 


Evaluate 

Engagement 
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Simulation  Model  Results 


#  of 

Incoming 

Threats 

Threat 

Interval 

(seconds) 

%of 

Successful 

First-time 

Engagements 

WIP 

[Saturation] 

(seconds) 

Average 
Total  Time 
(seconds) 

Limiting 
Process  #1 

Limiting 
Process 
#1  Time 
(seconds) 

Limiting 
Process  #2 

Limiting 
Process 
#2  Time 
(seconds) 

1 

60 

90 

0.5166 

25.1135 

"ID  the  Threat" 

0.3694 

"Establish  a 

Track" 

0.2561 

1 

60 

100 

0.4974 

23.4344 

"ID  the  Threat" 

0.3551 

"Establish  a 

Track" 

0.2648 

1 

30 

90 

1.2893 

34.8877 

"ID  the  Threat" 

0.872 

"Determine  an 
Engagement 
Solution" 

0.5203 

1 

30 

100 

1.2033 

32.1173 

"ID  the  Threat" 

0.8961 

"Establish  a 

Track" 

0.4846 

2 

60 

90 

2.3242 

63.4865 

"Transmit  All 

Data" 

2.0821 

"Compile 

Inventory 

Data" 

1.7562 

2 

60 

100 

2.2656 

59.8616 

"Transmit  All 

Data" 

2.3936 

"Compile 

Inventory 

Data" 

1.9017 

2 

30 

90 

5.8229 

120.15 

"Transmit  All 

Data" 

3.3926 

"Compile 

Inventory 

Data" 

3.0117 

2 

30 

100 

5.7013 

111.14 

"Transmit  All 

Data" 

3.7807 

"Compile 

Inventory 

Data" 

3.271 
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Revised  FORCEnet  OA  Model 


PEO  IWS  OA 
Warfare  System  / 
Functional  Domain 


Revised  FORCEnet 
OA  System  / 
Functional  Domain 


Detect 


Search  /  Detect 

Search  /  Detect 

•  Sensor  Asset 

•  Sensor  Asset 

•  Sensor  Report 

- N 

•  Sensor  Report 

•  Sensor  T rack  Report 

1 - iX 

•  Sensor  Track  Report 

•  INTEL  Report 

•  INTEL  Report 

•  Measurement  Report 

•  Measurement  Report 
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?  ErUATE  Revised  FORCEnet  OA  Model  (cont’d) 


PEO  IWS  OA 
Warfare  System  / 
Functional  Domain 


Revised  FORCEnet 
OA  System  / 
Functional  Domain 


Control 


Data  /  Information  Services 

•  Sensor  Track 

•  Supporting  Source  Track 

•  Classification 

•  Track  Kinematics 

•  Attribute  Data 

•  T rack  Repository 

•  NRT  INTEL  Track 

•  Sensor  Scheduler 


T 


Planning,  Assessment  &  Decision 

•  Assign  Missions 

•  Tactical  Picture 

•  Action  Plans 

•  Capability 

•  Plan 

•  Mission  Assessment 

•  Threat  Assessment 

•  C2  Order,  Schedule,  &  Event 


Data  /  Information  Services 

•  Sensor  Track 

•  Supporting  Source  Track 

•  Classification 

•  Track  Kinematics 

•  Attribute  Data 

•  Track  Repository 

•  NRT  INTEL  Track 


Planning,  Assessment  &  Decision 

•  Sensor  Scheduler 

•  Assign  Missions 

•  Tactical  Picture 

•  Action  Plans 

•  Capability 

•  Plan 

•  Mission  Assessment 

•  Threat  Assessment 

•  C2  Order,  Schedule,  &  Event 

•  Schedule:  Weapon,  RV, 
NAV  &  Engineering 

•  Action:  Weapon,  RV,  NAV 
&  Engineering 
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Revised  FORCEnet  OA  Model  (cont  d) 


PEO  IWS  OA 


Revised  FORCEnet 


Warfare  System  / 
Functional  Domain 


OA  System  / 
Functional  Domain 


Control 


▲ 


Weapon  /  Asset  Services 

•  Action:  Weapon,  RV,  NAV  & 
Engineering 

•  Schedule:  Weapon,  RV  & 
Engineering 

•  Event:  Weapon,  RV,  NAV,  & 
Engineering 


{to  Planning,  Assessment  &  Decision} 


{to  EXCOMM} 
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Revised  FORCEnet  OA  Model  (contd) 


PEO  IWS  OA 


Engage 


Warfare  System  / 
Functional  Domain 


Mission  Execution 

•  Air  /  Surface  Missile 

•  Land  Attack  Missile 

•  Torpedo 

•  Gun 

•  Decoy 

RV  Assets 

•  Aircraft 

•  Boat 

•  Un-manned  Vehicle 

RV  Assets 

•  Engineering 

•  Damage 

•  Bridge 


Revised  FORCEnet 
OA  System  / 
Functional  Domain 


Mission  Execution 

•  Air/ Surface  Missile 

•  Land  Attack  Missile 

•  Torpedo 

•  Gun 

•  Decoy 

RV  Assets 

•  Aircraft 

•  Boat 

•  Un-manned  Vehicle 

RV  Assets 

•  Engineering 

•  Damage 

•  Bridge 

Kill  Assessment 
Midcourse  Guidance 
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Revised  FORCEnet  OA  Model  (contd) 


PEO  IWS  OA 
Warfare  System  / 
Functional  Domain 


Revised  FORCEnet 
OA  System  / 
Functional  Domain 


Common  Services 

•  Display 

•  NAV 

•  Database 

•  Time 

•  DX / DR 

•  Environment 


Common  Services 

•  Display 

•  NAV 

•  Database 

•  Time 

•  DX / DR 

•  Environment 


EXCOMM 

•  Communications  Service 
Action 

•  Network  Schedule 

•  Message  Event 

•  Network 

•  Radios 

•  Data  Links 

•  SatComm 


EXCOMM 

•  Communications  Service 
Action 

•  Network  Schedule 

•  Message  Event 

•  Network 

•  Radios 

•  Data  Links 

•  SatComm 

•  BG  /BF  COA  Orders 

•  Event:  Weapon,  RV,  NAV 
&  Engineering 
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Revised  FORCEnet  OA  Functional  Model 


3.0  Planning,  Assessment, 
and  Decision  (PAD) 


1.0  Search  and  Detect  (SD) 


•  ID  Threat 

•  Select  Preferred  Sensor 

•  Verify  Threat  and  Intel 


4.0  EXCOMM 


•  Update  Network  (Threat  Info) 

•  Transmit  Data  (Threat) 

•  Request  G1  Data _ 

•  Receive  G1  Data 

•  Transmit  Data  (FCS) _ 

•  Receive  G1  Data_2 

•  Relay  Info  to  Supporting  Ship(s) 

•  Contact  Threat  Tracking  Ship 

•  Contact  Firing  Ship _ 

•  Transmit  Data  (Firing  Command) 

•  Compile  Data  (Inventory) _ 

•  Transmit  Data  (Inventory) 

•  Verify  Transmission  (Inventory) 

•  Transmit  Data  (Engagement) 

•  Transmit  All  Data 

•  Verify  Transmission  (All  Data) 


5.0  Mission  Execution  (ME) 


•  Prepare  Weapon 

•  Update  Inventory  of  Firing  Ship 

•  Uplink  Midcourse  Guidance 

•  Sched.  Sensor  for  Term.  Support 

•  Validate  Sensor  Track  Data 

•  Perform  Kill  Assessment 

•  Mark  as  Kill  or  Miss 

•  Compile  Engagement  Eval.  Data 

•  Compile  Inventory  Data 

•  Compile  Asset  Data 

•  Assess  Shared  Data 


•  Determine  Engagement  Solution 

•  Apply  Engagement  Doctrine 

•  Formulate  Solution 

•  Select  Weapon  Type 

•  Select  Ship  for  Sensor  Track 

•  Select  Ship  for  Weapon  Use 


5  J  E5  0  0 


6.0  Common  Services  (CS) 


•  Compile  Data  (Threat  Info) 

•  Verify  Transmission  (Threat  Info) 

•  Compile  Data  (FCS  Info) 

•  Verify  Transmission  (FCS  Info) 

•  Compile  Data  (Firing  Cmnd.  Info) 

•  Verify  Trans.  (Fir.  Cmnd.  Info) 

•  Compile  Data  (Engagement  Info) 

•  Verify  Trans.  (Engagement  Info) 
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-Revised  FORCEnet  OA  System  Domain 


1.0  Search  and  Detect  (SD) 


Sensor  Asset 


Sensor  Report  (Status) 


Sensor  Track  Report 


INTEL  Report 


Measurement  Report 


2.0  Track  Data  (TD) 


Ship  Track  (System  Track) 


Support  Source  Track 


Track  Repository 


Track  Kinematics 


Attribute  Data 


"J  A  ®  [|][|][|] 

a  a  ^ 


3.0  Planning,  Assessment, 
and  Decision  (PAD) 


•  Threat  Assessment 


Sensor  Scheduler 


Assigned  Missions 


Plan 


C2  Order,  Schedule,  Event 


Tactical  Picture 


Capability 


Schedule:  Weap.,  RV,  NAV  Engin. 


J  S  0  0 

a  a  v  v  v 


4.0  EXCOMM 


G1  Data 


G2  Data 


G3  Data 


Data  Links 


Network 


SATCOM 


Radios 


BG/BF/COA  Orders 


5.0  Mission  Execution  (ME) 


0  ST 


6.0  Common  Services  (CS) 


•  Display 


•  Time 


•  Navigation 


•  Environment 


•  Databases 


S  0  0 
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Conclusions  /  Recommendations 


•  Functional  analysis  of  PEO  IWS  OA  Functional 
Domain  model  resulted  in  revised  FORCEnet  OA 
model 


Simulation  validated  logical  functional  flow  of 
revised  FORCEnet  OA  model 

Simulation  validated  system  timing  of  revised 
FORCEnet  OA  model 

Revised  FORCEnet  OA  model  and  Simulation  used 
for  model  validation  merit  further  scrutiny  and 
refinement 
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Recommended  Future  Actions 


Incorporate  real-world  timing  parameters  of  current/future 
combat  systems 

Incorporate  real-world  threat  characteristics  and  parameters 
into  simulation  model 

Use  simulation  model  to  determine  threat  “processing”  time 
using 

Refine  simulation  model  based  on  results  from  using  real- 
world  combat  system  and  threat  parameters 

Use  refined  simulation  model  to  assess  combat  system 
network  architectures  for  saturation  nodes  and  choke  points 

Assess  simulation  model  against  other  likely  engagement 
scenarios 

Continue  to  revise  NAVSEA  FORCEnet  OA  model  for  use  in 
architecting  future  single  ship  and  strike  group  systems  and 
networks 
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Modeling  and  Simulation 
(M&S)  an  Issue  of  Credibility 

David  Henry 
Lockheed  Martin 
MS2-Moorestown 


11/1/2006 


NDIA  System  Engineering  9th 
Annual  Conference 


“The  Challenge” 


•  Develop  an  “acceptable”  representation  of 
reality 

•  Produce  desirable  and  acceptable  results 


How  does  the  M&S  developer/analyst  offer  reasonable  grounds  for  being  believed? 


11/1/2006 


NDIA  System  Engineering  9th 
Annual  Conference 
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Background 

•  Things  to  overcome:  (the  way  it  is) 

-  Traditionally  placed  in  the  engineering  tool  category 

-  The  benefit  is  at  the  task  level  (one  who  can  wield  the 
hammer  is  the  benefactor) 

-  The  “it-wasn’t-built-here”  syndrome 

•  Things  to  reach  for:  (the  way  to  go) 

-  Increase  to  primary  use  (reduce  cost,  reduce  risk) 

-  Re-use  (stop  re-inventing  a  wheel) 

-  Leverage  to  gain  greater  sophistication/complexity 


Takes  an  incredible  level  of  knowledge  to  transport  the  level  of  belief  needed. 
The  more  complex  the  more  knowledge  base  required. 
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Approach 

•  Move  from  a  tool  concept  to  a  knowledge 
base  concept 

•  Define  common  processes  and 
approaches  to  represent  consistent 
knowledge  levels 


Three  (3)  levels  of  representation: 

Registration,  Certification,  Accreditation 


11/1/2006 


NDIA  System  Engineering  9th 
Annual  Conference 


Registration 

•  Implement  common  practice  to  recognize 
models  (subsystem/engineering)  by  a 
specific  group/body  of  experts 

•  Institute  formal  process  (as  simple  as): 

-  Cataloguing  models 

-  Generating  Read-Me  files 

-  Performing  comparative  analysis  (similar 
algorithms/models) 


Easy  way  to  capture/present  models  of  various  types  in  multiple  categories 

THE  KEY  -  Common  Taxonomy 

11/1/2006  NDIA  System  Engineering  9th  5 

Annual  Conference 


Taxonomy  Structure  Example 


Hierarchy 
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Certification 


•  Clearly  and  fully  document  characteristics 
of  models/simulations  (platform/system) 
beyond  Read-Me  files/cataloguing 

•  Must  meet  verification  and  validation 
requirements  (customer  satisfaction) 

•  Demonstrates  adherence  to  a  credible  CM 
process 

Most  cases  certified  M&S  will  be  built  from  registered  M&S 
Clear  contribution  to  growth  of  maturity  of  M&S  at  this  level 
Larger  community  commitment  and  interest  -  Process  Bar  Raised 
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Accreditation 


•  Most  widely  recognized/implemented  quality 
assurance  method  (Well  Documented) 

“Accreditation:  the  official  determination  that  a  computer  model  is  acceptable  for  a 
specific  purpose.  [MORS]”. 

•  M&S  (platform/system  to  warfare/campaign) 
must  convey  reality  (as  much  as  possible) 

•  M&S  intent  of  use  is  clear  and  agreed  to  (official 
approval  process) 


Certification  efforts  direct  contribution  to  this  level 
Knowledge  level  boundary  crossed  from  “what”  is  represented  to  “how  well” 

11/1/2006  NDIA  System  Engineering  9th  8 
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Thought  For  the  Day! 


Broaden  the  approach  of  accreditation  of  models  and  simulations 
across  communities  (training,  analysis,  acquisition,  testing, 
planning,  experiment)  extending  the  specific  use  portion 
of  the  definition  to  a  different  level. 

The  statement  that  most  appropriately  applies  here  is  that  models 
and  simulations  are  rapidly  evolving  into  a  major  vehicle  through 
which  the  intuition  and  insight  regarding  combat  capabilities  will  be 
developed  and  proven. 


Will  drive  reuse  consideration/determination  early  in  the  planning  process 


11/1/2006 


NDIA  System  Engineering  9th 
Annual  Conference 


M&S  Potential  Use 


Gaming 


Environment  leads  to  reuse  -  true  enabler  of  knowledge  exchange 

1 1/1/2006  NDIA  System  Engineering  9th 
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Some  Notes  on  M&S  Potential  Use 


•  Experimentation  -  addresses  problems 
and  situations  that  can  not  be  done  any 
other  way  or  it  is  the  only  way 

•  Such  applications  could  reach  down  into 
Testing  and  Demonstrations 

•  Gaming  is  a  whole  new  perspective  from 
the  game  itself  to  being  a  “player”  of/in  it 

•  Such  applications  could  reach  down  into 
Training,  Testing,  and  Demonstrations 

To  “maintain”  certification  or  accreditation  models  and  simulations  must 
continue  to  be  well-developed,  well-documented,  well-calibrated 
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Conclusion 


•  Need  to  establish  additional  methods  to 
characterize  credibility 

•  Further  refinement  of  definitions  associated  with 
M&S  goodness  and  clearer  defined  roles  and 
responsibilities  (agents,  authorities,  review 
panels) 

•  The  emphasis  on  M&S  credibility  has  stimulated 
V&V  activities  to  begin  earlier  in  model 
development  and  part  of  the  M&S  life  cycle 

It  is  believed  that  other  industries  could  easily  adopt  this  strategy 
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System  Level  M&S  Context  (Hierarchy) 


M&S  Uses 

Concept  Exploration 
Requirements  Analysis 
Performance  Analysis 
Engineering  Design  Trades 
Engineering  Design 
Test  &  Evaluation 
System  Acceptance 


cf^p 


WARFARE 

MISSION 

AREA 


FORCE  ON  FORCE 
LEVEL 


PLATFORM/SYSTEM 

LEVEL 


Primary  Target 
Audience 

ustomer 


ystem  Architect 


System  Analyst 


SUBSYSTEM/ENGINEERING  LEVEL 


Engineering  Design 


Modeling  and  Simulation  Types 


Use  the  Right  M&S  Tool  For  The  Job 


Figure  2 
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Taxonomy  Value 

-  Basis  for  classification  of  analytical  modeling  and 
simulation  techniques 

-  Standardized  vernacular  that  may  be  used  to 
describe  these  techniques 

-  Stimulate  more  communication  between  the  various 
Analysis,  Modeling,  and  Simulation  communities 

-  Intended  to  be  framework  for  comparing  and 
contrasting  various  techniques 

-  Envisioned  as  a  standard  template  of  attributes  to  be 
addressed  in  descriptions  of  analytical  modeling  and 
simulation  efforts 
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How  to  Create  and  Select 
High  Value,  Innovative,  Competitive 
and  Economic  Concepts 
for  New  Systems? 
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Jacob  Herscovitz 
RAFAEL 

Space  Systems  Directorate 
Haifa,  Israel 


Dr.  Amihud  Hari  Prof.  Menachem  P.  Weiss 

Design  Speedovation  TECHNION 

Israel  Faculty  of  Mech.  Eng. 

Haifa,  Israel 
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_Jopics 

1 


•Introduction 
•Overview  of  ICDM 
•Case  Study 
•Research 

•Application  of  ICDM 
•Conclusions 
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The  Early  Phases  of  the  System  Life  Cycle  Process 
are  the  Most  Critical  to  the  Success  of  a 

New  System 

LCC  committed 


90% 


75% 


50% 


Failures  appear 


•Failure  Modes 

•^Therefore  the  use  of  an 

efficient  method  for  this 

created 

bridal  stage  is  most 

important 

more  than  75%  of  its  LCC  is  committed 


Manufacturing  Operation  phase 


Life  cycle 

Product  Concept  Ful 
Definition  Design  Development 

(Including 
Preliminary 
Design) 
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Common  difficulties  in  the 
Conceptual  Design  Phase: 


r 


No  adequate  team 
assigned  to  the  project 


Resources  not 
completely  allocated 

Vi 


Time  pressure 


Investments  are 
very  risky 


Evaluation  of  several 
concepts  in  parallel 


Decisions  making 
with  fuzzy 
information 


I  * 


Working  under  pressure . . . 
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RISKS  of  Inadequate  Conceptual  Design 


Selection  of  tht 
first  proposed 
concept 


Selection  of  the 
most  recently 
proposed  concept 


Dictated 

concept 


Superior  in  one 
criteria  (local 
optimization) 


Suppress 
creativity 
or  synthesis 


Missed 
critical  customer 
needs 


No 

consensus  and 
inefficient 
teamworl 
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RISKS  of  Inadequate  conceptual  design  (cont.) 

4 

Selection  of  poor  concept 

* 

Conceptual  changes  during  advanced  development 

t?  0  '4 

More  expenses  Delays  Inferior  products 

-il  [?  ^  t? 

Loss  of  profits  Loss  of  market  share 
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Topics 


♦Introduction _ 

•Overview  of  ICDM 

•Case  Study 
•Research 

•Application  of  ICDM 
•Conclusions 


FLAT  PLASMA 
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ICDM:  INTEGRATED  CUSTOMER  DRIVEN 
CONCEPTUAL  DESIGN  METHOD 


Covers  the 
entire  process  of  the 
conceptual  design 
and 

preliminary  design  stages 
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ICDM:  INTEGRATED  CUSTOMER  DRIVEN 
CONCEPTUAL  DESIGN  METHOD 


1.  Identification 

2.  Translation  of  the 

3.  Abstraction 

of  Customers 

Needs  into  Product 

Definition  of  the 

and  their  Needs 

Definition 

basic  problems 

* . 

. 

4.  Creation  of 

5 .  Designation  of  the 

6.  Synthesis  of 

solutions  to  the 

concept  Evaluation 

the  many  Primary 

basic  problems 

Criteria 

Concepts 

♦ 


* 


7.  Main 
Concepts 

* 

8.  Design  and 
Analysis  of  the 

9.  Final 
Concept 

10.  Launch 
of  the 

Selection 

Main  Concepts 

Selection 

Project 
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ICDM:  INTEGRATED  CUSTOMER  DRIVEN 
CONCEPTUAL  DESIGN  METHOD 


1 .  Identification 
of  Customers 
and  their  Needs 


4.  Creation  of 
solutions  to  the 
basic  problems 


3.  Abstraction 
inition  of  the 
sic  problems 


Integrates  and  modifies 
methods  and  techniques, 
provides  a  selection  of 
tools  ior 


6.  Synthesis  of 
the  many  Primary 
Concepts 


7.  Main 

y  each  step 

) 

51 

flilX 

J 

10.  Launch 

Concepts 

Selection 

Analysis  of  the 
Main  Concepts 

Concept 

Selection 

of  the 
Project 
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ICDM:  INTEGRATED  CUSTOMER  DRIVEN 
CONCEPTUAL  DESIGN  METHOD 


O  _ A  1.  —  a — _ JT _ 

j 

De 

FAST 

1 

ie 

basic  probien 

as 

Brain- 
Storming 
TRIZ 


5.  Designation  of  the 
concept  Evaluation 
Criteria 


Pugh 

Method 

ocictuui 

r 

CFMA 

9.  Final 

CDTC 

Concept 

RTA 

Selection 

CSDR 

Project 

October  25th.  2006 


J.  Herscovitz,  A.  Hari,  M.  P.  Weiss  -  9th  Annual  Systems  Engineering  Conference,  NDIA  2006 


12 


ICDM:  INTEGRATED  CUSTOMER  DRIVEN 
CONCEPTUAL  DESIGN  METHOD 

^Tailoring  Guidelines: 

ICDM  has  an  open  architecture,  is  flexible  and  can  be 
tailored  to  the  unique  needs  of  each  case  and  organization. 

*  The  Human  (soft)  Aspect: 

ICDM  Provides  the  means  to  improve  motivation  and 
creativity  of  new  product  development  teams 

^  Design  Quality  Measurement  (DQM) 

ICDM  includes  an  on-line  customer  value  based  DQM 
system 
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Topics 


1 

•Introduction 

•Overview  of  ICDM 

| - \ 

♦Case  Study _ m 

•Research 

•Application  of  ICDM 
•Conclusions 
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Step  1 :  Identification  of  Customers  and 

their  Needs 


Problem  Definition: 

Due  to  repetitive  cases  in  which  anti-terrorist  or  infantry 
forces  were  injured  by  rifle  shooting,  from  friendly  forces, 
a  need  emerges  for  an  identification  device.  This  device 
will  warn  the  shooter  when  he  aims  at  his  own  forces. 


Product  to  Design:  “PIFF2000  ” 

Personal  IFF  (Interrogator  of  Friend  or  Foe) 
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Customers 

The  users 

•  Infantry  combat  teams 

•  Anti-terror  squads 


Stakeholders  (Army): 

•  purchasing  authorities 

•  logistics  support  authorities 

•  combat  doctrine  authorities 
Stakeholders  (Civil): 

•  Investors 

•  Regulatory  &  environment  authorities 
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Customer  Needs 


Prevent  firearms 
shooting  on 
friendly  forces 


Operate  in  reference 
scenarios 


Easy  to  use 


Safe 


Works  reliably 


Maintenance 


Affordable 


Delight  the  customer 


Warn  shooter  from  friendly  force 


System  works  anywhere: 
(Urban  areas,  Dense  jungles, 
Wilderness) 


A  friendly  and  hostile  soldiers 
In  close  proximity  and  the 
system  discriminates _ 


For  shooter  and  target 


Does  not  interfere  with  combat  activity 

Low  weight 

Integrate  into  combat  gear 

Immune  to  spoofing 

From  shooting 

Does  not  scratch,  wound,  hurt  or  press 

No  false  alarms 

Works  when  needed 

Resistant  to  rain,  shocks,  dust,  moisture 

Alarms  when  removed  or  unsafe  install 

Simple  service 

Comprehensive  self  test 


Purchasing 


Service 


Integrate  in  C4I  networks 


Compass,  Navigation 
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Step  2:  Translation  of  the  Needs 
into  Product  Definition 


Performance  Based  Specification 


Decision  T  able 


Spec 

Weight 

Trade 

offs 

Wireless 

Binoculars 

Target! 

value 

^Mff. 

Cost 

TTM 

ConcepKI 

1 

False  Alarm 

Rate 

20.5 

N/A 

N/A 

2104 

• 

• 

O 

No 

2 

Optical 

Radiation 

11.9 

1 

0.0 

0.0 

0.5 

A 

A 

A 

Yes 

3 

Range 

11.3 

2 

comply 

comply 

1000.0 

A 

O 

O 

Yes 

4 

Resolution 

11.1 

bad 

bad 

20.0 

• 

• 

O 

Yes 

5 

Autonomy 

8.7 

yes 

yes 

yes 

A 

A 

A 

No 

6 

EM  Radiation 

8.4 

1,3 

Over 

spec. 

0.0 

0.2 

A 

A 

A 

No 

7 

Cont. 

Operation 

7.9 

1,2,6 

20.0 

20.0 

24.0 

• 

• 

A 

No 

8 

Volume 

5.4 

7,4 

large 

500.0 

200.0 

O 

A 

A 

No 

9 

Price 

4.9 

1,2 

expensive 

cheap 

750.0 

O 

O 

O 

No 

10 

Weight 

4.2 

7,8 

large 

500.0 

200.0 

o 

O 

A 

No 

11 

Maintenance 

3.8 

9,5,2 

moderate 

cheap 

50.0 

1° 

A 

A 

No 

12 

Mounting  timei 

2.1 

8,9,11 

N/A 

N/A 

5.0 

A 

A 

No 

- ^ - ' 

v - y  ^ ^ 
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Step  3:  Abstraction,  Definition  of  the  Basic  Problems 


How  to 

achieve 

resolution 


What  type  of 

communication 

from  shooter 
to  target 

9 


How  to 

display 
interrogation 
results 

9 


Which 

operation 
mode  and 
model 

9 


Coexistence 

if  multiple  system; 
in  the  arena: 

How? 


How  and  where 

to  install 
the  devices 

9 
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STEP  4:  CREATING  SOLUTIONS  TO 
THE  BASIC  PROBLEMS 


Problem 


Solution 


.  <VVV\V 


Cv 


Resolution 

ElectroOptics 

5 

5 

++ 

ElectroMagnetics 

2 

2 

- 

GPS 

5 

3 

+ 

Triangulation  with 

3 

3 

+ 

directional  ant. 

Communication 

Radio 

3 

5 

++ 

Type 

Microwave 

5 

3 

++ 

Optics  (laser) 

5 

5 

++ 

Display 

Graphics  LCD 

5 

3 

++ 

LED 

5 

5 

++ 

Flag 

5 

3 

++ 

Speaker 

2 

5 

+ 

Buzzer 

2 

5 

+ 

Vibrator 

2 

3 

- 
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STEP  5:  EVALUATION  CRITERIA 


# 

Criteria 

A 

B 

Target  value  (QFD) 

1 

Technology  gaps 

+ 

+ 

2 

Mission  reliability 

+ 

0.95 

3 

FAR 

+ 

210-4 

4 

Operation  range 

+ 

+ 

1000  m 

5 

Soldier  suitability 

+ 

+ 

6 

Radiation 

+ 

+ 

5e-7  mj/cm2 

7 

Resolution 

+ 

+ 

20  m 

8 

Operation  cost 

+ 

50  $/month 

9 

Volume/  weight 

+ 

200  cc  /  200  g  -  target 
200  cc  /  200  g  -  shooter 

10 

Continuous  operation 

+ 

24  hours 

11 

Cost 

+ 

750$  total  (250+500) 

12 

Manufacturing  capability 

+ 

+ 

13 

Installation,  removal  and 
ease  of  use 

+ 

+ 

2  min  preparation 

14 

Customer  attraction 

+ 

15 

Serviceability 

+ 

+ 
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Step  6:  Synthesis  of  Primary  Concepts 


Morphological 
Table  Tool 


Grades  Legend: 
[Performance,  Lack  of  risks] 
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Step  7 :  Evaluation  of  the  primary  PIFF 
concepts  using  PUGH  method 


Concept 

Criterion 

Local 

Triangulation 

Cellular 

Triangulation 

Optical 

Helmet  - 

Sensors 

Optical 

Helmet  - 

Fisheye 

Optical 

Helmet  - 

Fibers 

Optical 

Helmet  - 

Radio 

Centralized 

GPS 

Distributed 

GPS 

Technology  gaps 

D 

s 

S 

S 

S 

S 

s 

s 

Performance: 

Operation  range 

A 

s 

- 

s 

s 

Soldier  suitability 

s 

+ 

+ 

+ 

+ 

s 

+  + 

Radiation 

T 

s 

- 

s 

s 

Resolution 

- 

+  + 

+  + 

+  + 

+  + 

+ 

+ 

Manufacture  capability 

U 

s 

+ 

+ 

+ 

+ 

s 

+ 

Inst.,  rem.  and  ease  of  use 

s 

- 

- 

- 

- 

s 

+ 

Serviceability 

M 

s 

s 

s 

s 

+ 

s 

+ 

1+ 

0 

4 

4 

4 

5 

1 

6 

I- 

1 

5 

5 

5 

3 

0 

0 

Total 

-1 

-1 

-1 

-1 

2 

1 

6 
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Local  Triangulation 


Optical  Helmet  - 
Fibers 
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The  Final  Concepts 


Cellular 

Triangulation 


Optical  Helmet  - 
Sensors 


Optical  Helmet  - 
RADIO 


Centralized  GPS 


Optical  Helmet  - 
Fisheye 


Distributed  GPS 
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ICDM:  INTEGRATED  CUSTOMER  DRIVEN 
CONCEPTUAL  DESIGN  METHOD 


October  25th.  2006 


8.  Design  and 
Analysis  of  the 
Main  Concepts 


J.  Herscovitz,  A.  Hari,  M.  P.  Weiss  -  9th  Annual  Systems  Engineering  Conference,  NDIA  2006 


26 


Conceptual  Architecture  and  Operation  Logic  Design 


Scenario 

Shooter  aims  at  target  and  interrogates 
If  FRIEND,  target  responds  to  shooter 
If  no  response,  then  target  is  FOE 


Shooter 


Target 


Resp 


onse 


Basic  Principles  of  IFF  System 


Shooter 


Target 

Interrogation  Response 

Architectural  Concept  in  Distributed  GPS  PIFF 
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Distributed  GPS  IFF  -  Principle  of  Operation 


Birget_Azimuth 
Range 


~  Rifle_azimuth) 
n  then  Friend 


Otherwise,  Foe 


Compass  derived  data 
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Mechanical  Conceptual  Design 


Rifle  Scope 

Radio  Ant. 

Elect.  Compass 

GPS  Ant. 

Inclinometer 

Elect.  Cards 

Elect.  Cover 

Batt.  Cover 

Batteries 

Interrogate  Lever 
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Station 


Legend: 

Transmit 

Receive 


1  . 


Electronics  Principle 


Time 


Satellite  link 


Radio  link 
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Manufacturing  and 
Assembly  concept 


Installation  concept 


Operation  and  Usage  concept 


Personnel  - 
skills 

requirements 
and  training 
policy 


Conceptual  Operation 

and  ILS  Decisions 


Verification,  Validation 
Maintenance  policy  and  Test  concept 


Transportation, 
handling  and 
packaging 
concept 


Safety  requirements 


Documentation 

concept 


Disassembly 
and  Recycling 
concept 
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CFMA  Table  for  “Interrogation”  Function 


— 

Failure 

mode 

Failure 

Result 

S 

Failure 

causes 

F 

Ways  to 
detect 

D 

n 

Corrective  action  A  iNewl 

- — _ ^/|sfd| 

No 

Transmission 

Mission 

Failure 

8 

Faulty 

Trans. 

4 

Test  first  batch 

I 

% 

160 

Continuous  self  test:  F=1 
Env.  Test  for  prototype: 
)=2 

16 

V  V7 


f c 
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Improved  Design  After  CFMA 


Insert  REDUNDANT  mechanisms: 

•  Original:  Geolocation  with  GPS 

•  Add:  Optics  interrogation  with  Radio  response 

•  Combined  logic:  perform  an  AND  check  from  both  sub-systems. 
Result:  Improvement  in  RELIABILITY  (both  Type  1  and  Type  2) 
Improved  redundant  performance: 

•  In  urban  areas,  GPS  may  not  receive  adequately.  Laser  will  perform. 

•  In  open  battlefield,  Laser  beam  may  be  obstructed  by  smoke,  dust,  etc. 
GPS  will  perform. 
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Improved  Concept 
Principle  -  Add  Self-Test 


Laser  Transmitter 


Sensor  Self-Test 


On  Helmet 


Light  Wave 


.  v 


Laser  Diode 
Transmitter 


Laser  Driver 


Sensor 


Preamp, 
►  AGC, 
Decoder 


Laser  Self-Test 


Sensor 


Preamp  & 
Level 
Decoder 


Laser  Diode 
Transmitter 


Laser  Driver 


Satellite  link 


GPS 

Antenna 


GPS 

Receiver 


On  Rifle 


Pitch 

Inclinometer 

Electronic 

Compass 

Interrogation 
/  Button 


Micro 

Processor 


Radio  link 


Radio 

Antenna 


Digital 

Transciever 


Display 


Encoder/ 

Decoder 


I - 1  On/Off  “ 

l«TERR  J3AT  ■ 


Power 

Supplies 
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Cost  Estimation 


built  from 


built  from 


Z4 

Main  Target 
Unit  Assembly 


5.0 


built  from 

2.2.5 

Logic  and 
Processing  Ci... 

30.0 


built  from 


built  from 


2.2.6 


Power  Supply 


20.0 


built  from 


2.2.8 


Main  Board 
Assembly 


15.0 


Date: 

Friday  12  January  200 

Author: 

1  Trial  User 

Number: 

2 

Name: 

(Trial)  Target  Unit 
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IS 


Sizing  -  Helps  to  measure 

and  optimise  the  specific 
functional  characteristics  of 
the  alternative  concepts 


oUZE 


20 


i 

i 

i 

i 

40 

60 

80 

100 

CFMA  -  Conceptual 
Failure  Modes  Analysis:  Helps 

to  identify  and  prevent  known 
and  potential  failures  from 
reaching  the  customer. 


120 


140 


Methods  for  Analysis,  Improvement  and 
Evaluation  of  the  Alternative  Concepts 


A&T 

I 

A&T 

8$ 

_= 

Case 

A&T  - 

CDT  C  —  Conceptual 
Design  to  Cost:  Helps  to 
decide  on  target  costs  and  to 
evaluate  costs  of  the 
alternative  concepts 


LCD 
Display  6$ 


"Fm3ef"PCB~” 

Assy.  27$ 


Case  & 

Parts 


hinder  PUB 

Assy.  27$ 
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Ease  & 

Parts 


RTA-Risk  &  Time  to 
Market  Analysis:  Helps  to 

define  and  analyze  Risks,  Plan 
the  development  process  and 
evaluate  the  TTM  of  the 
alternative  concepts 


A&T  - 

A&T  - 

Trans 
ceiver  5$ 

Duple 
xer  5$ 

PAL 

8$ 

Print  & 

Misc. 

Comp. 

Trans 
ceiver  5$ 

Duple 
xer  5$ 

PAL 

8$ 

Print  & 

Misc. 

Comp. 
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mm 


Step  9:  Final  Concept  Evaluation 


Spec 

Weight 

Target  value 

Result 

Achieved 

CSR 

1 

FAR 

20.5 

2-KH 

10-6 

100 

2 

Radiation  .Opt 

11.9 

510'5  [J/cm2] 

<  15  pJ/cm2 

100 

3 

Range 

11.3 

1000  [m] 

1000 

100 

4 

Resolution 

11.1 

20  [m] 

<  20  m 

100 

5 

Autonomy 

8.7 

Full 

Full 

100 

6 

Radiation  .EM 

8.4 

0.2  fw/cm2] 

0.024 

100 

7 

Cont.  Operation 

7.9 

24  [h] 

20.0 

60 

8 

Volume 

5.4 

200  [cm3] 

245 

70 

9 

Price 

4.9 

750  [$] 

454 

100 

10 

Weight 

4.2 

200  [g] 

200 

100 

11 

Maintenance 

3.8 

50  [$/month] 

50 

100 

12 

Mount  /  Dismount 

2.1 

5  [min] 

<5 

100 

Total:  94.5% 
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Step  10:  Project  Launch 


Goals: 

1 .  To  approve  the  specification  and  the  concept 

2.  To  approve  the  resources  required  for  the  full 
scale  development. 


A  A 


JAr  Vv 
□□ 

When  complex  system  is  involved,  step  10  serves 
as  the  starting  point  of  the  next  lower  tier  in  the 
Systems  Engineering  Process  (SEP). 
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Topics 


•Introduction 
•Overview  of  ICDM 
•Case  Study _ 

♦Research _ 

•Application  of  ICDM 
•Conclusions 
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29  new  product  development  teams  (129  members)  were 
instructed  to  apply  the  ICDM  methodology  step  by  step  and 
evaluate  the  results  by  the  DOM  (Design  Quality  measurement) 
system. 


Which  approach  created  higher  CSR 

(better  concepts) 

Intuitive  Concept 

or 

concept  created  by  using  ICDM 

9 
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Total  CSR 


O  The  quality  of  a  concept  achieved  by  ICDM  was  on  the 
average  1.75  times  better  than  the  intuitive  alternative. 


generic  ICDM 
improvement  tools 
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The  Paradigm  Shift  (1) 


The  Traditional  Paradigm: 
Intuitive  concepts  are  the  best 


l 

The  New  Paradigm: 
Methodology  can  help  in 
creating  better  concepts 
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What  types  of  metrics  were  used  by 

most  of  the  teams 


□  Safety 

□  Functional 

□  Design  level  /  aesthetics 

□  Interface  and  compatibility 

□  Environment  Friendliness 

□  Manufacturability 


□Cost 

□Time  to  market  /  Risk 
□Reliability 

□Maintainability  &  Support 
□Friendliness 
□Operation  time 
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©  The  5  metrics  that  were  used  by  most  of  the  teams 
covered,  on  average,  more  than  80%  of  the  customer 
satisfaction. 


Average  Weight  of  Product  Characteristics 


All  others 
17.0% 


Time  to 
market  / 
Risk 
13.5% 


Cost 

13.6% 


Functional 

23.1% 

Friendliness 
18.1% 


Reliability 

14.7% 
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•Which  type  of  metrics  achieved  the  highest 
Customer  Satisfaction  Rating  (CSR)  ? 

•Which  type  of  metrics  achieved  the  lowest 
Customer  Satisfaction  Rating  (CSR)  ? 


□Safety 

□Functional 

□Design  level  /  aesthetics 
□Interface  and  compatibility 
□Environment  Friendliness 
□Manufacturability 
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©  The  teams  achieved  the  best  scores  for  the  functional  and  safety 
metrics  on  account  of  other  Product  Characteristics  like: 
user  friendliness,  reliability,  cost  and  time  to  market. 


Average  CSR 


50%  60%  70%  80%  90%  100% 


Safety 

Functional 
Design  level  /  aesthetics 
Maintainability  &  Support 
Time  to  market  /  Risk 
Friendliness 
Operation  time 
Interface  and  compatibility 

Reliability 
Environment  Friendliness 
Manufacturability 
Cost 


%99.0 
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O  The  Design  Quality  Measurement  system  detected 

cost,  reliability  and  risk  problems  very  early  in 
the  design  process,  when  it  is  relatively  easy  to 
correct  them  and  prevent  them  from  reaching  the 
customers. 


©  The  research  revealed  that  the  generic  tool  of 
Conceptual  Failure  Mode  Analysis  (CFMA) 
reduced  the  criticality  of  the  selected  concept 
by  more  than  50%. 


October  25th.  2006 
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The  Paradigm  Shift  (2) 


The  Traditional  Paradigm:  Cost, 
reliability  and  risk  problems 
can  be  solved  only  during  the 
Full  Scale  Development  (FSD) 
phase 

1 

The  New  Paradigm:  We  can 
detect ,  prevent  and  correct 

cost,  reliability,  and  risk 
problems  during  the  Conceptual 
Design  Phase  (CDP) 


October  25th.  2006 
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Topics 

1 - 

•Introduction 
•Overview  of  ICDM 
•Case  Study 
•Research 

(  \ 

•Application  of  ICDM 

v _ _ _ _ _ _ _ _ _ | 

•Conclusions 
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Special  Application  of  ICDM 

Formal  SE  requires  time  and  resources  which  are  not 
always  available  and  therefore  SE  processes  are 
sometimes  totally  skipped 

Two  options  for  short  and  cost  effective  processes 


•Agile  workshop  (Agile  SE):  2-3  Month 


Steps  1,2 

Steps  3-7 

Step  8 

Step  9 

Step  10 

1-4  weeks 

2-5  days 

1  -6  weeks 

1-2 

days 

2  half 
days 

•  Slim  workshop  (Slim  SE):  5  Days  !!! 


Day  1: 

Day  2: 

Day  3: 

Day  4: 

Day  5: 

Needs 

Benchmark 

Solutions 

Design 

Selection 

and 

and 

and 

and 

and 

Requirements 

Abstraction 

Concepts 

Analysis 

Verification 
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•  One  day  ICDM  overview 

•Two  days  workshop  on  new  product  definition  (NPD) 
and  Conceptual  Design 

•  Two  days  workshop  on  ICDM  methodology  and 
tools 

•  5  days  class  action  learning  workshop  -  ICDM 
applied  on  the  company  project 

•  One  semester  academic  graduate  and 
undergraduate  course  on  ICDM 
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ICDM  is  a  proven  method  for  performing  the 
critical  task  of  conceptual  design  in  the  SE 
process  and  to  make  the  best  choice. 


CDM  -  Integrated  Customer  Driven 
Conceptual  Design  Method. 


\  WIRELESS  LAN 
ADSL  MODULE 
GIVES  BROADBAND 
INTERNET  CONNECTION 
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Thank  You,  Questions? 


Jacob  Herscovitz 

Space  Systems  Directorate 
RAFAEL 

Tel:  +972-4-879-2379 
Email:  jacobh@rafael.co.il 


October  25th.  2006 
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Interdisciplinary  Team 
Project  Management  Using 
TSPsm  Concepts 

Prepared  For: 

NDIA  Systems  Engineering  Conference 

Oct.  25,  2006 


Prepared  By: 

Bradley  Hodgins 

NAVAIR  Systems/Software  Support  Center  (NSSC) 


Presentation  Objectives 


•  Background 

-  NAVAIR  and  SPIKE 

-  Personal  Software  Process  (PSP)  and 
Team  Software  Process  (TSP)1 

•  SPIKE  Project  Planning 

-  Project  Results  (before  and  after) 

-  Initial  Project  Planning 

-  Project  Execution  (day-to-day) 


Personal  Software  Process,  PSP,  Team  Software  Process,  and  TSP  are  service  marks  of  Carnegie  Mellon  University 

nav^Tai  r 


NAVAIR  Software/Systems  Support  Center  (NSSC) 
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Who  is  NAVAIR? 


•  NAVAIR  is  the  Naval  Air  Systems  Command. 

•  We  develop,  acquire,  and  support  the  aircraft  and  related 
weapons  systems  used  by  the  U.  S.  Navy  and  Marine  Corps. 

•  We  translate  the  needs  of  the  Navy  and  Marine  Corps  into  the 
technical  and  financial  requirements  needed  by  industry  to 
actually  produce  an  aircraft  or  other  weapon  system. 

•  Our  goal  is  to  provide  the  fleet  with  quality  products  that  are  both 
affordable  and  available  when  they  are  most  needed. 

•  Our  support  extends  across  the  entire  life  span  of  a  product, 
including  all  upgrades  and  modifications  to  that  product. 
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Where  is  NAVAIR? 


China  Lake 

WEAPONS 
DIVISION 


Pt  Mugu 

WEAPONS 

DIVISION 


Lakehurst 

ALRE -  SUPPORT  EQ 
AIRCRAFT  DIVISION 


Patuxent  River 

NAVAIRHQ,  PEOs 
AIRCRAFT  DIVISION 


NAVAIR  Headquarters 
O  Acquisition/Test/Development  Centers 


Orlando 

TRAINING 

SYSTEMS 

DIVISION 
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Who  is  SPIKE? 


•  SPIKE  Project 
Office  is  part  of 
NAVAIR  Weapons 
Division  at  China 
Lake,  Ca. 

•  Developing  a  low- 
cost,  lightweight 
guided  weapon  for 
U.S.  ground  forces. 

•  Using  TSP  in  their  software  projects  since  August  2002. 
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Accomplishments 

Guided  Missile  Field  Test 


12  April  2006  at  NAVAIR  Weapons  Division 
Target  Range:  1000  meters 
Target  Diameter :  2  meter 
Impact:  8  inches  from  center 

Missile  bore  sighted  35  meters  left  and  105 
meters  above  the  target 
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Project 

Management 


Who  is  SPIKE? 


SPIKE  Organizational  Structure 


Systems 

Engineer 


(part-time  manager) 


Tech 


Mech 


Team  Leader  and 
Team  Members 
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What  is  PSP/TSP? 

•  PSP  shows  software  professionals  how  to 

-  plan  and  track  their  personal  work 

-  define  processes  that  best  suit  them 

-  measure  and  manage  cost,  schedule,  and  quality 

•  TSP  shows  teams  of  PSP-trained  professionals 
how  to 

-  establish  realistic  commitments 

-  keep  management  informed 

-  deliver  quality  products 

-  minimize  project  cost  and  schedule 


NAVAIR  Software/Systems  Support  Center  (NSSC) 
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PSP/TSP  Benefits 

•  PSP/TSP  quickly  improves  the  performance  of  software 
groups. 

•  Planning  and  tracking  is  accurate,  timely,  and  precise. 

•  Product  quality  is  managed  and  measured  from  the 
beginning  of  the  job. 


•  By  finding  and  fixing  problems  before  test,  project 
cycle  time  is  substantially  reduced. 


NAVAIR  Software/Systems  Support  Center  (NSSC) 
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Typical  TSP  Launch 

Review  Management  goals  and  project  objectives. 
Set  Software  Team  goals. 

Produce  development  strategy  and  process. 

Produce  top-down  plan. 

Review  quality  plan. 

Produce  bottom-up  plan  (detailed  individual  plans). 
Perform  risk  assessment. 

Brief  plan  to  Management. 
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SPIKE  Team  Planning  Meeting 


Review  Management  goals  and  project  objectives. 

Set  Software  Team  goals. 

Produce  development  strategy  and  process. 

Produce  top-down  plan. 

Review  quality  plan. 

Produce  bottom-up  plan  (detailed  individual  plans). 
Perform  risk  assessment. 

Brief  plan  to  Management. 
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SPIKE  Team  Project  Results 


Project 

Cycle 

Completion 

Date 

Planned 

Weeks 

Actual 

Weeks 

Difference 
in  Weeks 

%  Error 

Final  Test 

i 

03/2003 

26 

29 

3 

+12% 

Successful 

2 

05/2004 

47 

60 

13 

+28% 

Successful 

3 

10/2005 

15 

19 

4 

+27% 

Successful 

4 

04/2006 

12 

14 

2 

+17% 

Successful 

5 

09/2006 

15 

15 

0 

0% 

Successful1 

Cycles  1,  2,  and  3:  Software  Team  used  TSP 


Cycles  4  and  5:  Software  Team  used  TSP; 

SPIKE  Team  Planning  Meeting  using  TSP  concepts 

'COTS  hardware  failure  during  Final  Test  caused  some  test  criteria  to  be  inconclusive 
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SPIKE  T earn  Planning  Meeting 


•  Who  should  be  there  (or  not)? 

•  How  to  talk  to  the  whole  system? 

•  What  are  the  required  activities? 

•  How  detailed  should  the  planning  be? 

•  What  are  the  task-to-task  dependencies? 


NAVAIR  Software/Systems  Support  Center  (NSSC) 
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Who  Should  Be  There? 

•  Everyone  in  the  trenches  (Producers) 

-  Hardware,  Software,  Mechanical,  Systems, 
Algorithms,  Technical  Support 

•  Get  buy-in  from  producers 

•  Whole  system  solutions 

•No  one  from  high-level  management 

-  Detract  from  the  details 
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How  To  Talk  To  The  Whole  System? 

•  Create  diagram  of  the  existing  system 

•  “red  line”  diagram  with  changes  and 
additions 

•  Discuss  every  block  on  the  diagram 


-  Identify  tasks  needed 
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Conceptual  Design  -  Before 
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How  To  Talk  To  The  Whole  System? 

•  Create  diagram  of  the  existing  system 

•  “red  line”  diagram  with  changes  and 
additions 

•  Discuss  every  block  on  the  diagram 

-  Identify  tasks  needed 

•  Let  discussion  go  where  it  goes 

•  Take  notes  on  the  side 

-  Lots  of  data  (on  and  off  the  board) 

-  Lessons  learned  from  previous  efforts 

-  Logistics  (material)  needs 


NAVAIR  Software/Systems  Support  Center  (NSSC) 


Slide  18 


NAV^TaI  R 


How  Detailed  Should  The  Planning  Be? 


•  Only  S/W  team  had  formal  PSP  training 

•  Two-pass  approach  to  estimating  tasks 


-1st  pass:  Task  assigned  to  Engineer 

Engineer  gives  Small-Medium-Large 
-  2nd  pass:  Engineer  gives  “number  of  days/weeks” 


•  Estimates  are  challenged  real-time  to  ensure 
realistic  basis  for  task  sizes 


NAVAIR  Software/Systems  Support  Center  (NSSC) 
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What  Are  The  Required  Activities? 


•  It’s  not  just  about  software  anymore 

•  Sample  workflow  for  creating  multi-layer  circuit  board 
(involves  EE,  ME,  S/W  Engr,  others) 


Design  circuits 

6  wks 

Test  design  (through  simulation) 

4  wks 

Lay  out  design  (w/  inspections) 

4-6  wks 

Manufacture  blank  circuit  board 

4  wks  (of  waiting) 

Inspect  board 

1  wk 

Get  board  “stuffed”  (attach  components) 

2  wks 

Inspect  board 

1  wk 

Debug  board 

2-4  wks 

NAVAIR  Software/Systems  Support  Center  (NSSC) 
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What  Are  The  Task-to-Task  Dependencies? 


Every  task  is  given  a  Project  Priority  and  a 
Milestone  Priority 

Project  Priorities 

-  1  -  High  -  Critical  to  Next  Firing  —  Have  to  Do 

-  2  -  Medium  -  Not  Critical  to  Next  Firing  —  Should  Do 

-  3  -  Low  -  Not  Critical  to  Next  Firing  —  Ought  to  Do 
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What  Are  The  Task-to-Task  Dependencies? 


•  Within  each  Project  Priority,  tasks  are 
categorized  into  Milestone  Priorities 

•  Milestone  Priorities 


-  #1  -  Now 

-  #2  -  After  now 

-  #3  -  For  Bench  Integration  Testing 

-  #4  -  For  Carco  Table  Testing 

-  #5  -  For  Live  Fire  Testing 


NAVAIR  Software/Systems  Support  Center  (NSSC) 


Slide  22 


NAV^TaI  R 


Wrapping  Up  Planning  Session 


•  Once  all  tasks  are  assigned,  sized,  and  given 
priorities,  load  leveling  is  performed. 

•  Risks  are  identified 


-  Impact  an  event  would  have  on  the  end  goal 

-  Systems  Engineer  tracks  these  during 
development 

•  Team  members  all  give  a  “thumbs-up”  that 
they  are  good  with  the  plan 

•  Plan  is  briefed  to  management  by  Systems 
Engineer 
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Project  Execution  (day-to-day) 


•  Collecting/reporting  status 

•  Tracking  progress 

•  Dealing  with  problems 


•  Everybody  has  the  task  list  from  the  SPIKE 
Team  Planning  Meeting 
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Collecting/Reporting  Status 

•  Systems  Engineer  meets  with  Team  Leads 
every  other  week 

•  On  the  off  weeks,  Engineers  send  in  their 
status  to  Systems  Engineer 


-  Refer  to  the  task  list 


-  Use  a  common  template  (<  5  mins  to  fill  out) 

-  Statuses  are  forwarded  to  Senior  Management 

•  6  weeks  before  live  fire  testing,  daily 
standup  meetings  are  held  (15  mins) 

-  Focuses  on  the  task  list 
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Tracking  Progress 

•  Systems  Engineer  uses  a  paper  copy  of  the 
task  list 


-  Task  list  sorted  by  priorities 

-  Tasks  marked  off  as  they  are  completed 


Before 


After 
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Dealing  With  Problems 


•  Systems  Engineer  is  the  problem  solver 


-  Tracks  risks  identified  during  project  planning 
meeting 

-  Looks  for  scheduling  problems  and  tasking 
conflicts 


-  Resolves  deadlocks 


-  Encourages  interdisciplinary  communication 


NAVAIR  Software/Systems  Support  Center  (NSSC) 
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Summary  Conclusions 

•  Weaknesses 

-  Hard  to  estimate  intangibles 

-  Shared  resources  are  still  sometimes  beyond  your 
control  (Carco  Table  and  Test  Ranges) 

•  Strengths 

-  Low-tech  tracking  is  easy  to  use  (and  works!) 

-  Everyone  knows  the  big  picture  (and  what  they  are 
responsible  for) 

-  Everyone  talks  the  same  language  (common 
vocabulary) 
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Contact  Information 

•  Dan  Crabtree  (SPIKE  Systems  Engineer) 

-  phone:  (760)  939-4406 

-  e-mail:  daniel.crabtree@navy.mil 

•Vic  Walkling  (SPIKE  Software  Lead) 

-  phone:  (760)  939-1592 

-  e-mail:  victor.walkling@navy.mil 

•  Brad  Hodgins  (NAVAIR  TSP  Coach  supporting  SPIKE ) 

-  phone:  (760)  939-0666/4446 

-  e-mail:  bradley.hodgins@navy.mil 
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Abbreviations 

•  NAVAIR  -  Naval  Air  Systems  Command 

•  NS  SC  -  NAVAIR  Systems  Software  Support  Center 

•  PSP  -  Personal  Software  Process 

•  SEI  -  Software  Engineering  Institute 

•  TSP  -  Team  Software  Process 
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•Mandatory 

•  All  Classified  Data  -  NISPOM/DoD  5220. 22-M,  (National  Industrial  Security 
Protection  Operating  Manual)  regulations 

-  Available  at  http://www.dss.mil/isec/nispom.htm,  141  pages 

-  Also  see  5220. 22-R  at  http://www.cfisac.org/FSQ%20Librarv/Misc/ISR- 
DRAFT.html 

•  SAP/SAR  Data  -  JAFAN  6/3  (Joint  Air  Force  -  Army  -  Navy)  Manual 
regulations 

-  FOUO,  162  pages 

•  DIA  SCI  Data  -  JDCSISSS  (Joint  DoDIIS/Cryptologic  SCI  Information 
Systems  Security  Standards)  regulations 

-  FOUO  but  can  be  found  on  the  Internet,  1 1 1  pages 

-  Available  at  http://www.fas.org/irp/doddir/dod/idcsisss-rev2.doc 
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•Mandatory 

•  CIA  SCI  Data  -  DCID  6/3  (Director  of  Central  Intelligence  Directive) 
regulations 

-  Flow  down  requirement  from  JDCSISSS  regulation 

-  FOUO  but  can  be  found  on  the  Internet,  ~90  pages 

-  Available  at  http://www.fas.org/irp/offdocs/DCID  6-3  20Manual.htm 

•  IT  systems  on  the  GIG  must  comply  with  DoD  8500  regulations 

-  Available  at  http://iase.disa.miI/policv.html#8500s 

•  “Defense-In-Depth  IA  Computer  Network  Defense”  CJCSM  6510.01 
Manual  regulations 

-  Flow  down  requirement  of  8500  regulations,  354  pages 

-  Available  at  www.namrl.navy.mil/FORMS/CJCSM  651001  .pdf 
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•Mandatory 

•  Exports  with  Military  Use  -  ITAR  (International  Traffic  in  Arms  Regulations) 

•  Exports  with  Commercial  Use  -  EAR  (Export  Administration  Regulations) 

•  National  Security  Systems  -  CNSS  (The  Committee  for  National  Security 
Systems)  regulations 

-  Available  at  http://www.cnss.gov 

-  See  http://csrc.nist.gov/publications/nistpubs/800-59/SP800-59.pdf  for 
criteria  for  identifying  National  Security  Systems 
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For  Each  DoD  Service 


3rd  Dimension 
for  each 
Protocol  or 
unique  HLA 
implementation 


Valid  AF-ICE  Data 
Labeling  Combinations  for 
Acquisition  Centers 
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National  Information  Assurance  (IA)  Glossary,  CNSS  Instruction  No  4009 

•  Identification  is  the  process  an  Information  system  uses  to  recognize  an  entity. 

•  Authentication  is  the  security  measure  designed  to  establish  the  validity  of  a 
transmission,  message,  or  originator,  or  a  means  of  verifying  an  individual's 
authorization  to  receive  specific  categories  of  information. 

•  Authorization  is  the  access  privileges  granted  to  a  user,  program,  or  process. 

•  Nonrepudiation  is  the  assurance  the  sender  of  data  is  provided  with  proof  of 
delivery  and  the  recipient  is  provided  with  proof  of  the  sender's  identity,  so  neither 
can  later  deny  having  processed  the  data. 

-  Accounting  is  the  tracing  of  information  activities  to  a  responsible  source. 

•  Confidentiality  is  the  assurance  that  information  is  not  disclosed  to  unauthorized 
individuals,  processes,  or  devices. 

•  Integrity  is  the  quality  of  an  information  system  reflecting  the  logical  correctness  and 
reliability  of  the  operating  system;  the  logical  completeness  of  the  hardware  and 
software  implementing  the  protection  mechanisms;  and  the  consistency  of  the  data 
structures  and  occurrence  of  the  stored  data.  Note  that,  in  a  formal  security  mode, 
integrity  is  interpreted  more  narrowly  to  mean  protection  against  unauthorized 
modification  or  destruction  of  information. 

•  Availability  is  the  timely,  reliable  access  to  data  and  information  services  for 
authorized  users. 
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•  CJCSI  621 2.01  D  “Interoperability  and  Supportability  of  Information 
Technology  and  National  Security  Systems”  (8  March  2006) 


1.  Joint  Interoperability  Test  Command  (JITC)  Joint  Interoperability  Test  and  Certification 
Policy: 

•  IT  system  meets  operational  needs 

•  Interoperable  with  existing  and  proposed  IT  and  National  Security  Systems  (NSS) 

•  Supportable  over  existing  and  planned  Global  Information  Grid  (GIG) 

•  Are  interoperable  with  allies  and  coalition  partners 

•  Are  net-ready 

•  Allow  US  forces  to  protect  mission  essential  data;  detect  and  respond  to  network 
intrusion/system  compromise;  and  restore  mission  essential  data 

2.  Net  Ready  Key  Performance  Parameters  (NR-KPPs)  are  a  mandatory  element  of  Joint 
Capability  Integration  and  Development  System  (JCIDS)  Capability  Development 
Documents  (CDDs)  and  Capability  Production  Documents  (CPDs)  and  Information 
Support  Plans  (ISPs).  Four  J-6  defined  NR-KPP  elements: 

•  Compliance  with  the  Net-Centric  Operations  and  Warfare  Model  (NCOW-RM) 

•  DoDAF  Integrated  Architecture  Products 

•  Compliance  with  GIG  Key  Interface  Profiles  (KIPs) 

•  Verification  and  compliance  with  DoD  information  assurance  requirements  (8500 
series  regulations  and  DIACAP/DITSCAP  accreditation  process) 
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HTTP/S 

•  Supports  the  “Admission  Control  Security  Model”  understood  by  all  CIO  Security 
Officers  for  providing  identification,  authentication,  authorization  and  accounting 
services 

•  Allows  both  the  resource  holder  and  enterprise  security  administrator  to  set  explicit 
access  control  privileges  to  maintain  confidentiality  of  specific  Web  resources 

HTTP/S  and  SIP/S  sessions  are  authenticated  before  being  authorized 

•  RFC  2617  uses  challenges  involving  private  keys 

-  To  authenticate  clients  or  consumers 

-  To  optionally  allow  the  client  to  authenticate  the  server  to  provide  mutual 
authentication 

•  TLS  encryption 

-  Requires  servers  or  producers  to  have  X.509  PKI  certificates 

-  Since  client  certificates  are  optional,  RFC  2617  may  still  be  used  to  authenticate 
the  client  to  provide  mutual  authentication 

Authorization  Mechanisms 

•  Identity  Based  Access  Controls  (IBAC),  includes  Access  Control  Lists  (ACL) 

•  Role  Based  Access  Controls  (RBAC) 

•  Attribute  Based  Access  Controls  (ABAC) 
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•  Clearance  is  eligibility,  it  requires  background  checks  sponsored  by  a 
government  agency  to  determine  if  resources  can  be  trusted  for  holding  or 
processing  classified  information. 

•  Authorization  is  determined  and  enforced  at  the  enterprise-level  often 
using  Mandatory  Access  Controls  (MAC),  especially  for  multi-security 
domain  transfers.  Regular  system  users  are  not  to  be  trusted  to  enforce 
authorization  policies.  The  ISSM,  NSO  or  ISSO  will  set  the  DoD 
authorization  policies  contained  within  the  AIS  system  Security  Support 
Structure  (SSS). 

•  Need-to-know  is  determined  and  enforced  by  the  Data  Holder  of  the 
information  with  Discretionary  Access  Controls  (DAC),  e.g.,  Identity  Based 
Access  Controls  (IBAC),  Access  Control  List  (ACL)  and  Role  Based  Access 
Controls  (RBAC)  or  need-to-know  is  established  by  the  Information  Owner 
and  enforced  by  RBAC. 

-  Note,  an  exclusive  need-to-know  RBAC  mechanism  requires  a  firmly  established 
OV-7  “Logical  Data  Model”  to  demonstrate  traceability  to  a  role’s  operational 
need  to  access  data.  OV-7  diagrams  are  easier  to  develop  for  C2  systems  than 
modeling  and  simulation  networks,  because  data  tends  to  belong  to  fewer  data 
owners  and  processes  are  static  and  are  not  as  intellectually  complex. 
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Protection  Levels 

PL1  (Dedicated  Mode) 

•  All  resources  (e.g.,  users,  systems...)  have  proper  clearance,  formal  access  approval 
and  need-to-know 

•  Most  of  today’s  modeling  and  simulation  resources  natively  operate  in  a  PL1 
environment  even  when  interfacing  to  MLS  guards 

PL2  (System  High  Mode) 

•  All  resources  have  proper  clearance  and  formal  access  approval 

•  Not  all  resources  have  need-to-know 

•  Requires  principals  to  authenticate  themselves  before  they  can  access  resources 

PL3  (Compartmented  Mode) 

•  All  resources  have  proper  clearance 

•  Not  all  resources  have  formal  access  approval 

•  Not  all  resources  have  need  to  know 

•  Requires  principals  to  authenticate  themselves  before  they  can  access  resources 

PL4  and  PL5  (Multi-level  Modes) 

•  True  Multi-Level  Security  (MLS) 

•  Requires  principals  to  authenticate  themselves  before  they  can  access  resources 
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•Network  security  requires  a  Network  Operation  Security  Center  (NOSC)  to 
perform  PL2  service-level  routing  and  PL3  transaction-level  authorization  services 
to  persist  physical  connectivity  to  mitigate  the  pains  of  establishing  event  specific 
MOAs. 


•Application-level  security  is  not  part  of  DIS  nor  HLA  and  is  new  to  TENA. 

•  Application-level  Identification  &  Authentication  (l&A)  must  be  explicit,  not  PL1 
implicit,  to  persistent  network  connections  in  PL2+  environments 

•  Application-level  Labeling  must  be  explicit,  not  PL1  or  PL2  implicit,  to  foster 
persistent  network  connections  in  PL3+  environments 

•  Simulation  architecture  needs  to  accommodate  proxy-based  Policy  Enforcement 
Points  (PEP)  and  Policy  Decision  Points  (PDP)  to  implement  mediated  net-centric 
access  controlled  security  services 

•Data  aggregation  (event  OPSEC)  challenges  are  driven  by  Security 
Classification  Guidelines  (SCG)  and  Program  Security  Directives  (PSD).  Data 
aggregation  will  remain  a  manual  Systems  Engineering  process  until  strong  data 
management,  explicit  labeling  and  IA  minded  compositioning  processes  are 
established.  Simulation  resources  sensitive  to  data  aggregation  must  have  event- 
level  access  controls  to  mitigate  chances  of  unauthorized  disclosures  to 
aggregated  information. 
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Protecting  IP  on  Classified  Networks 

•  Identities  must  be  established  to  prevent  unauthorized  disclosure 

-  Identities  can  be  established  local  to  the  private  enterprise 

-  Identities  can  be  established  and  asserted  by  a  trusted  3rd  party 

•  Accounts  are  provisioned  internal  to  the  enterprises  owning  the  IP 

•  Auditing  will  be  similar  to  PL2  auditing  requirements 

•  Authentication  will  be  mutual  similar  to  PL4  authentication  requirements 

•  Authorization  to  IP  must  be  implemented  by  private  enterprises  owning  the 
IP 

•  WAN  communications  for  IP  within  NSA  Type  I  encrypted  tunnels  must  be 
encrypted  once  again  using  industrial  strength  hop-by-hop  or  end-to-end 
commercial  encryption  (e.g.,  FIPS  140-2)  similar  to  non  Sources  and 
Methods  Information  (SAMI)  PL3  encryption  mechanisms 
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•  Supports  the  “Admission  Control  Security  Model”  and  DoD  PL3+  Confidentiality  services 

•  RFC  2617  digest-based  challenge  produced  by  a  proxy  server  or  user  agent  prompts  the 
user  with  an  identity-based  challenge  and  realm 

•  HTTP/S  proxy  server  acts  as  a  Policy  Enforcement  Point  (PEP) 


User  supplies  the  appropriate 
username  and  password 

Unfortunately  HTTP/S  can’t 
provide  real-time  application 
services  because  there  is  not 
a  separation  between  the 
real-time  streams  and  Request 
controlling  transactions  *GET* 

To  scale,  distributed  M&S 
synchronous  real-time 
streaming  applications  require 
asynchronous  peer-to-peer 
(P2P)  communication, 
preferably  using  a  proven  Web 
user  agent  technology 


HTTP 


Web  Browser 
User  Agent 


Server 
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SIP  Modeling  &  Simulation  User  Agents  (UA) 

•  Is  the  convergence  of  scalable  VoIP  phone  technology 

•  SIP  UAs  have  URIs  associated  with  them  that  are  routed  at  the  session-layer  to 
establish,  modify  and  tear-down  SIP  sessions 

•  Once  sessions  are  established  SIP  UAs  are  the  sources  and  sinks  of  real-time 
information,  typically  Real-time  Protocol  (RTP)  streams 

-  RTP  is  a  Web  standard  like  SIP  and  RTP  can  carry  DIS  payloads 

-  RTP  is  built  on  top  of  UDP 

SIP  UA  Integration  with  DIS 

•  User  agents  can  be  integrated  with  DIS  simulations  by  one  of  three  means: 

-  UAs  act  as  dedicated  very  loosely  coupled  hardware  relays  on  separate  hosts 

-  UAs  are  loosely  coupled  software  shims  that  are  installed  on  existing 
simulation  hosts 

»  capture  broadcast  traffic  with  a  virtual  network  adapter 

»  user  agent  listens  on  that  port,  bundles  traffic  in  RTP  packets  and  relays  them  via  RTP 
sessions 


-  UA  APIs  support  two  forms  of  tight  coupling  using  an  API 
»  Tightly  integrated  RTP  stack  with  loosely  coupled  SIP  stack 
»  Tightly  integrated  RTP  and  SIP  stacks 


jjfcu iSsiiu  Tia-il.-Ti  ma  rasw 

lr 


* 

mm 


Gestalt 


Streaming  UA 
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User  Agent 
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User  Agent  Proxy  &  DNS  Firewall 

Registrar 
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Real-Time  SIP 
Streaming  UA 
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•  SIP  network  resources  (C2  systems,  simulators,  etc.)  are  treated  as  opaque  URIs 

•  SIP  components  may  be  integrated  into  a  Multi  Channel  Service  Oriented 
Architecture  (MCSOA)  with  Web  Service  Description  Language  (WSDL)  interfaces 

•  SIP  supports  both  perimeter  and  application-level  security  models  for  real-time 
applications 

•  SIP  URI  resources  can  have  privileges  (e.g.,  IBAC,  RBAC  or  ABAC)  associated 
with  them  to  implement  discretionary  or  mandatory  access  controls 

•  SIP  can  utilize  RFC  2617  or  TLS  authentication  mechanisms  that  can  take 
advantage  of  LDAP,  RADIUS,  or  Diameter  services 

-  These  technologies  are  understood  and  trusted  by  many  CIO  Security  Officers 

•  SIP  supports  stateful  NAT  firewalls  used  by  most  enterprises  to  protect  intellectual 
property  and  export  controlled  information 

•  SIP  allows  existing  enclaves  to  be  added  into  larger  networks  without  requiring  IP 

address  re-alignments 

•  SIP  can  support  persistent  PL2+  networks  without  requiring  re-accreditation  of 
networks,  PL1  rated  systems  require  frequent  re-accreditation 

-  Zero  network  configuration 

•  SIP  is  strong  candidate  for  providing  common  integrated  wire-level  net-centric 
Security  and  Quality  of  Service  (QoS)  capabilities  to  DIS,  HLA  and  TENA 
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Extra  Slides 


Process  Steps  Below  Must  Be  Neutral  to  the  DIS,  TENA  and  HLA  Standards 
and  Workflow  Services  are  Innate  to  all  Seven  Steps 

1.  Formulate  Event 

2.  Specify  Battlespace 

•  Scheduling/Reservation  Services 

•  Discovery  Services 

3.  Build  Battlespace 

•  Composition  Services 

•  Orchestration  Services 

-  Resource  Provisioning  Services 

4.  Integrate  Battlespace 

•  Registration  Services 

•  Presence  Services 

•  Availability  Services 

5.  Execute  Event 

•  Resolve  Resource  Services 

•  Distributed  Dynamic  Networks  Services 

•  Monitoring  Services 

•  Execution  Services 

•  Network  Bandwidth  Throttling  and  Packet  Latency  Services 


6.  Analyze  and  Report  on  Event 

7.  Sustain  Event  Documentation  and  Environment 
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Single  Enclave  View 
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ODUSD(l&E)  in  AT&L  Role 


Acquisition,  Technology  and  Logistics 


Base  Realignment  &  Closure 

Business  Enterprise  Integration 

Environmental  Management 

Environmental  Readiness  & 
Safety 

Emerging  Contaminants 

Housing  &  Competitive 
Sourcing 

Installation  Requirements  & 
Management 

DoD  Explosives  Safety  Board 
Pest  Management  Board 


•  Manage  DoD  facilities  and 
infrastructure 

•  Defense  Facilities  Strategic  Plan 

•  Key  metrics  to  provide  quality 
facilities  and  infrastructure  that 
directly  support  mission  needs 
and  readiness 

•  Manage  Natural  Infrastructure 
assets  at  installations  and  ranges 
in  an  environmentally  sound 
manner  to  prevent  encroachment 
and  support  military  force 
operational  needs 


3 


ODUSD(  I  &E)  Acquisition 
ESOH  Leadership 

Acquisition,  Technology  and  Logistics 


DSOC  Task  Forces 


4 


ODUSD(l&E)  Role  in 
Acquisition 

Acquisition,  Technology  and  Logistics 

•  DAB/ITAB  Special  Advisor  for  ESOH  issues 

•  Oversight  of  AC  AT  ID,  IAM,  and  Special 
Interest  programs 

•  Focus  on  DoD  5000  —  ESOH  in  acquisition 
policy  development 

•  Develop  guidance  for  acquisition  ESOH  policy 

•  ESOH  input  to  CJCS  3170.01  series  -  JCIDS 


ODUSDO&E)  Role  in 
Acquisition 

Acquisition,  Technology  and  Logistics 

•  Chair  DoD  Acquisition  ESOH  IPT 

-  Consensus  on  ESOH  policy  and  guidance 

-  Influence  NSS  Acquisition  Policy  and  JCIDS 

•  Member  of  the  DAP WG/DAPSG 

-  DoDI  5000.2  and  Defense  Acquisition  Guidebook 

-  Acquisition  Community  Connection  (ACC)  site 

•  Member  of  Defense  Safety  Oversight  Council 

-  Co-chair  DSOC  Integration  Group 

-  Acquisition  and  Technology  Programs  Task  Force 


ODUSD(l&E)  Perspective 

Acquisition,  Technology  and  Logistics 

•  Why  be  concerned  with  ESOH  in  Acquisition? 

•  ESOH  considerations  affect  the  operational 

effectiveness  and  sustainability  of  the  system 

-  There  is  a  relationship  between  the  natural 
infrastructure  and  the  military  mission 

-  Compliance  requirements  and  encroachment  influence 
how  DoD  maintains  and  trains  with  the  system 

-  System  design,  operation,  and  maintenance  parameters 
determine  the  installation  and  workforce  needs  to  train 
and  maintain  the  system 
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ODUSD(l&E)  Perspective 

Acquisition,  Technology  and  Logistics 

•  The  goal  is  to  identify  life-cycle  ESOH  risks  early 
and  influence  system  design,  not  address  them 
afterwards  as  operational  considerations 

•  E,  S,  and  OH  considerations  are  inter-related  and 
should  be  assessed  holistically 

•  ESOH  hazards  are  best  addressed  using  a 
rigorous,  analytical  process 

•  System  design  is  most  effectively  influenced 
through  the  system  engineering  (SE)  process 
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ODUSD(l&E)  Perspective 

Acquisition,  Technology  and  Logistics 


•  ESOH  risks  include: 

-  Hazardous  materials  and  waste 

-  Environmental  and  occupational  noise 

-  Personnel  safety  and  occupational  health 

-  Regulatory  compliance 

-  System  safety  and  explosives  safety 

•  Need  to  manage  ESOH  risks  associated  with: 

-  Routine  operation  and  maintenance  of  the  system 

-  System  failures 

-  ESOH  compliance  requirements 


ESOH  Policy  Requirements 

Acquisition,  Technology  and  Logistics 

•  Top  level  principles: 

-  Address  safety  throughout  the  acquisition  process 

-  Use  a  total  systems  approach  to  minimize  or  eliminate 
characteristics  that  produce  environmental,  safety  or 
health  hazards,  where  practicable  and  cost  effective 

-  Use  the  system  safety  methodology  to  minimize 
ESOH  hazards  where  possible  and  manage  ESOH 
risks  where  they  cannot  be  avoided 

-  Accept  risks  at  designated  management  level 

-  During  system  design,  document  HAZMAT  in  the 
system  and  plan  for  demilitarization/disposal 
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ESOH  Policy  Requirements 

Acquisition,  Technology  and  Logistics 

•  Programmatic  ESOH  Evaluation  (PESHE) 

-  Updated  at  major  milestones  and  must  document: 

•  Strategy  for  integrating  ESOH  considerations 
into  the  systems  engineering  process 

•  Identification  of  ESOH  responsibilities 

•  Identification  of  ESOH  risks 

•  Method  for  tracking  progress  in  the 
management  and  mitigation  of  ESOH  risks 

•  NEP A/EO  12114  Compliance  Schedule 
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ESOH  Policy  Requirements 

Acquisition,  Technology  and  Logistics 

•  USD(AT&L)  Memorandum,  Defense  Acquisition 
System  Safety,  23  September  2004 

-  DoD  Standard  Practice  for  System  Safety,  MIL-STD- 
882D,  must  be  used  in  all  developmental  and 
sustaining  engineering  activities 

-  Systems  Engineering  Plan  must  include  the  strategy  to 
integrate  ESOH  risk  management  into  the  systems 
engineering  and  overall  risk  management  processes 

-  ESOH  risk  status  and  acceptance  decisions  must  be 

reported  at  technical  reviews  and  in  the  Program 
Review  Process  12 


Acquisition  ESOH  Activities 

Acquisition,  Technology  and  Logistics 

•  Program  oversight  and  assistance 

•  ESOH  risk  acceptance  policy 

•  Guidance  -  DAG  and  ACC 

•  Acquisition  education  and  training  -  DAU 

•  System  safety  process  -  MIL-STD-882 

•  System  safety-ESOH  management  evaluation 
criteria 
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Activities:  Oversight 

Acquisition,  Technology  and  Logistics 

•  Participate  in  MS  and  program  review  process 

-  Attend  program  WIPT/OIPT/DAB 

-  Review  documents  for  ESOH  coverage 

•  PESHE  and  Acquisition  Strategy 

-  Exploring  cooperation  on  DAPS  and  SEP  Reviews 

-  Work  with  programs  to 

•  Clarify  DoD  5000  ESOH  policy  requirements 

•  Emphasize  integration  of  ESOH  into  SE 

•  Focus  on  ESOH  risk  management 


Activities:  Risk  Acceptance 

Acquisition,  Technology  and  Logistics 

•  Draft  revision  to  system-related  ESOH  risk 
acceptance  policy  to  clarify  existing  DoD  system- 
related  ESOH  risk  acceptance  requirements 

-  Requires  the  user  representative  to  be  part  of  the 
ESOH  risk  acceptance  process 

-  Requires  the  user  representative  to  formally  concur 
with  all  Serious  and  High  ESOH  risk  acceptance 
decisions 

-  Requires  formal  acceptance  of  ESOH  risks  prior  to 

exposing  people,  equipment,  or  the  environment  to  a 
known  system-related  ESOH  hazard  15 


Activities:  Guidance 


Acquisition,  Technology  and  Logistics 


•  Defense  Acquisition  Guidebook 

-  System  Safety  Analyses  added  as  Input  and  Output  on 
systems  engineering  Vee-charts 

-  Incorporate  ESOH  steps/activities  into  Section  4.3 
Systems  Engineering  Activities  and  4.4  Design 
Considerations 

-  Update  Section  4.4. 1 1  ESOH 

•  Acquisition  Community  Connection  (ACC) 

-  Update  and  expand  ESOH  Special  Interest  Area 
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Activities:  DAU  Curricula 


Acquisition,  Technology  and  Logistics 


•  System  Safety  in  Systems  Engineering 
Continuous  Learning  Module  (April  2005) 

-  Identifies  System  Safety  activities  supporting  each  of 
the  SE  activities  in  each  phase  of  a  systems  life  cycle 

-  Course  (CLE009)  available  on-line 
http://www.dau.mil/basedocs/continuousleaming.asp 

•  Conduct  review  of  prioritized  courses  and  provide 
ESOH  input  consistent  with  current  policy 

-  Acquisition  core,  SPRDE,  Program  Management, 
Acquisition  Logistics,  and  Test  and  Evaluation 
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Activities:  MIL-STD-882 


Acquisition,  Technology  and  Logistics 


•  DoD  Acquisition  ESOH  IPT  had  concerns  with 
“E”  version  and  agreed  to  revise 

-  More  clearly  reflect  DoD  intent  of  document 

-  Connect  to  all  segments  of  SE  and  ESOH 

-  Clarify  "ESOH"  is  an  acronym,  not  a  discipline 

-  Return  Section  4  (only)  to  882D  language  and  revise 
only  as  necessary  for  currency  with  DoD  policy 

-  Include  some  tasks  as  non-mandatory  guidance 

-  Standardize  risk  definitions 
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Activities:  Evaluation  Criteria 

Acquisition,  Technology  and  Logistics 

•  Focus  on  assessing  overall  management  of 
System  Safety  -  ESOH  considerations  as  a  part  of 
the  SE  process  -  not  specific  ESOH  risks 

•  Established  four  evaluation  categories 

-  ESOH  Planning 

-  ESOH  Hazard  Identification,  Analysis,  and  Risk 
Acceptance 

-  ESOH  Requirements  for  the  System  and  Associated 
Infrastructure 

-  Personnel  and  Funding  for  ESOH 


Activities:  Evaluation  Criteria 

Acquisition,  Technology  and  Logistics 

•  Metrics  for  the  four  evaluation  categories  were 
developed  for  each  acquisition  phase 

•  Selected  key  activities  or  documentation  that 
would  be  a  strong  indicator  of  a  system  safety  (E, 
S,  and  OH)  for  each  acquisition  phase,  and  rated 
them  as  Green,  Yellow,  or  Red  (G/Y/R) 

•  Weighted  each  category  metric  for  summing  to  a 
single  overall  rating  (G/Y/R)  for  each  life  cycle 
phase 
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Activities:  Evaluation  Criteria 

Four  Categories  covered  in  O&S  Phase 

Acquisition,  Technology  and  Logistics 


ESOH  Planning  ESOH  Hazard  ESOH  Requirements  Personnel  and 

Identification,  for  the  System  and  Funding  for  ESOH 

Analysis,  and  Risk  Associated 

Acceptance  Infrastructure 


What  are  the  mishap  rates  for 
class  B  and  C  mishaps  during 
the  reporting  period,  and 
how  many  class  A  mishaps 
for  the  system  or  subsystem 
occurred  during  the  current 
calendar  year? 

Green  -  No  class  A  mishaps; 
no  increase  in  mishap  rates 
for  either  class  B  or  C  as 
compared  to  the  prior 
calendar  year 

Yellow  -  No  class  A  mishaps; 
Mishap  rate  increasing  for 
either  class  B  or  C  mishaps  as 
compared  to  the  prior 
calendar  year. 

Red  -  One  or  more  class  A 
mishaps  reported  in  the 
current  calendar  year 
class  A  mishaps  reported. 


What  is  the  highest  risk 
category,  are  there  any 
system  level  hazards  with 
formally  accepted  high  risks, 
and  are  there  any  system 
level  hazards  without  formal 
risk  acceptance? 

Green  -  No  hazards  with 
formally  accepted  high  risks 
and  no  hazards  without 
formal  risk  acceptance 

Yellow  -One  or  more  hazards 
with  formally  accepted  high 
risks,  or  any  hazards  with 
medium  and  low  risks  that 
have  not  been  formally 
accepted 

Red  -  One  or  more  hazards 
with  serious  or  high  risks  that 
have  not  been  formally 
accepted 


How  many  open  technical 
data  change  requests  (e.g. 
Technical  Orders,  Technical 
Manuals,  etc.)  have  been 
submitted  through  the  formal 
technical  data  change  system 
to  resolve  hazardous  material 
or  safety  issues  for  the 
system? 

Green  -  All  open  requests 
were  received  during  the  last 
six  months 

Yellow  -  One  or  more 
requests  has  been  open  for  six 
-12  months 

Red  -  one  or  more  requests 
have  remained  open  for  more 
than  1  year. 


What  is  the  level  of  effort  (LOE) 
in  man-years  (recurring) 
expended  by  the  program 
(organic,  matrix,  and  contract) 
for  environment,  safety,  and 
occupational  health  (ESOH) 
management? 

Green  -  constant  LOE  compared 
to  the  prior  fiscal  year. 

Yellow  -  decreasing  LOE 
compared  to  the  prior  fiscal  year. 

Red  -  zero  LOE 
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Activities:  Evaluation  Criteria 

ESOH  Hazard  Identification,  Analysis,  and  Risk  Acceptance  Category  across  all  Phases 

Acquisition,  Technology  and  Logistics 


Concept  Refinement  Technology  System  Development  Production  and  Operations  and 

Development  and  Demonstration  Deployment  Support 


Is  there  a  Preliminary 
Hazard  List  (PHL) 
developed  for  each  concept 
and  is  it  used  in  developing 
the  Analysis  of  Alternatives 
(AoA)? 

Green  -  Yes 

Yellow  -  Incomplete  PHL  or 
complete  PHL,  but  not  used 
to  influence  the  AoA 

Red  -  No 


Does  the  updated  PHL 
evaluate  enabling/  critical 
technologies? 

Green  -  Yes 

Yellow  -  Some,  but  not  all,  of 
the  enabling/  critical 
technologies  have  been 
assessed  for  ESOH  hazards 

Red  -  No 


Are  the  appropriate  levels  of 
hazard  analyses  completed 
and  presented  at  each  major 
design  review?  For  example, 
is  the  Preliminary  Hazard 
Analysis  (PHA)  completed 
and  status  of  hazards 
presented  at  Preliminary 
Design  Review  (PDR),  the 
majority  of  hazard  analyses 
completed  and  presented  at 
Critical  Design  Review 
(CDR),  and  status  of  ESOH 
risks  presented  at  Production 
Readiness  Review 
(PRR)  /System  Verification 
Review  (SVR)? 

Green  -  Yes 


Has  the  program  (1) 
continued  to  evaluate  the 
system's  test  and  operational 
performance  to  identify  new 
hazards,  (2)  continued  to 
track  all  hazards,  and  (3) 
obtained  formal  acceptance, 
at  the  appropriate 
management  levels,  of  all 
residual  ESOH  risks  and 
communicated  those  risks  to 
the  receiving  activities? 

Green  -  Yes 

Yellow  -  Satisfying  two  of 
the  three  criteria 

Red  -  Satisfying  one  or  none 
of  the  three  criteria 


What  is  the  highest  risk 
category,  are  there  any 
system  level  hazards  with 
formally  accepted  high  risks, 
and  are  there  any  system 
level  hazards  without  formal 
risk  acceptance? 

Green  -  No  hazards  with 
formally  accepted  high  risks 
and  no  hazards  without 
formal  risk  acceptance 

Yellow  -One  or  more 
hazards  with  formally 
accepted  high  risks,  or  any 
hazards  with  medium  and 
low  risks  that  have  not  been 
formally  accepted 


Yellow  -  Not  all  the 
necessary  hazard  analyses 
have  been  completed, 
and/ or  presented  at  the 
design  reviews 


Red  -  One  or  more  hazards 
with  serious  or  high  risks 
that  have  not  been  formally 
accepted 


Red  -  No,  hazard  analyses 
have  not  been  completed  in 
time  to  influence  the  design 
review  process 


Activities:  Evaluation  Criteria 

Acquisition,  Technology  and  Logistics 

•  Published  an  OSD  Guide  version  1 .0 

-  Technical  and  Program  Reviews  (self  assessment) 

-  Milestone  Review  Process  (oversight  assessment) 

•  Working  with  AT&L  SSE  to  incorporate  into  the 
Defense  Acquisition  Program  Support  (DAPS)  SE 
Assessment  Methodology 

•  Include  in: 

-  Acquisition  Community  Connection,  Defense 
Acquisition  Guidebook,  DAU  curricula 
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Acquisition,  Technology  and  Logistics 


Questions? 


Trish  Huheey 

ODUSD(I&E)  Acquisition  ESOH  Lead 

(703)  604-1846 
patricia.huheey@osd.mil 


ODUSD(l&E)  Perspective 

Acquisition,  Technology  and  Logistics 

•  Integrating  ESOH  considerations  into  systems 
engineering  early  in  the  acquisition  process: 

-  Protects  military  and  civilian  personnel  by  reducing 
hazards/risks  to  personnel  and  equipment 

-  Reduces  accidents  proactively 

-  Improves  warfighting  capability  and  combat  readiness 

-  Reduces  total  ownership  costs 

-  Lowers  risk  of  environmental  damage 

-  Prioritizes  hazards  for  corrective  action 

-  Reduces  need  for  system  retrofits 


Activities: 


•  MIL-STD-882D,  FEBOO, 
DoD  Standard  Practice  for 
System  Safety 

•  Use  of  “D”  version  is 
mandatory  for  DoD 

•  Product  of  Specs  &  Stds 
reform 

-  7  of  26  pages  mandatory 

-  Prescribes  “What”  not 
“How” 

-  Not  popular  with  many 
System  Safety  practitioners 


Ml  L-  STD-882 


Acquisition,  Technology  and  Logistics 

•  Preparing  Activity  sent 
out  final  draft  “E”  version 
out  for  informal  review 

-  Significant  departure  from 
MIL-STD-882D 

-  G-48  “E”  version  seeks  to 
restore  lost  guidance  on 
"How"  from  "C"  rejected 
during  Acquisition  Reform 

•  DoD  Acquisition  ESOH 
IPT  asked  to  review  by 
Lead  Standardization 
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Activity 


Interactive  Visualization 

Development 

For  Rapid  Response 


Therese  Metcalf 
October  2006 


Approved  for  Public  Release;  Distribution  Unlimited  -  06-1271 
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Overview  -  Modified  Systems 
Development  Approach 


■  Accelerated  system  engineering  process 
presented  through  an  example  development  effort 
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Concept  -  Problem/Solution 


Concept 


Problem:  No  global  view  of  Computer 
Network  Defense  (CND)  status  -  Results  in 
limited  data  for  decision  making  and  lack  of 
integrated  defense  tactics 

-  Limited  data  presentation  integration 

Fragmented  situational  awareness 


Current  data  is  disparate  and 
presented  in  a  tabular  format 
limits  quick  assessment 


Need  for  overall  understanding 
Lack  of  global  visualization 


Differing  products  present 
their  unique,  but  restricted 
view  of  the  global  situation 


Capabilities  not  available  in  Commercial- 
Off-The-Shelf  (COTS)  software 
“The  [display]  requirement  has  been  around 
since  day  one  of  the  mission  inception 


Basic  Data  (Data  Points) 

1  *  Information  (integrated  display) 

*  Knowledge  (layered  Information) 
1 — *  Understanding  (scenarios) 

1  *  Decision  Making  (response) 


Solution:  Interactive  three  dimensional  (3D)  Graphical  User  Interface  (GUI)  display 
presenting  a  fusion  of  actual  NetD  attack  status  in  real-time  across  the  cyber 
domain,  enhancing  user  perception  of  dense  defensive  Information  Warfare  (IW) 
data  for  situational  awareness,  quick  assessment,  decision-making,  and  immediate 

response  to  experienced  threat  activity 


Requirements  -  Tracking 


Requirements 


■  #:  Requirement  tracking  number 

■  Requirements:  A  brief  description 

White:  Open  Requirements 
Gray:  No  longer  relevant,  OBE 

-  Blue:  Details  TBD,  Priority  5 

-  Green:  Implemented  &  Tested 

■  Suggestion:  description  of 
enhancement  provided  by  a  user  - 

■  Modules:  Requirement  applied  to 

■  Related  Modules:  Potential  module(s) 
effected 

■  Priority:  Original  &  New  Priority 

-  1:  Immediate,  next  release 

-  2:  Following  release 

3:  Will  be  completed  in  designated 
timeframe  if  enough  resources 

-  4:  Will  investigate  possibility 
5:  Will  consider  in  the  future 

D:  Delivered  in  operational  prototype 


■  Source:  Requirement  submitter 

UD:  User  Derived  Requirement 

DD:  Developer  Derived  Requirement, 
may  be  suggestions  from  the  field 
or  the  Display  team 

■  Status  Type:  The  status  of  the 
suggested  enhancement 

Reviewed:  By  Display  Team, 
disposition  indicated 

Evaluation:  Under  evaluation  by 
Display  Team 

Pending:  Evaluation  by  Display 
Team 

■  Disposition: 

■  Release:  Version  requirement  will 
be  included  within 

■  Questions/Comments 

■  Query:  Query  tool  required 

■  Wks:  Weeks  to  work 


MITRE 
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User  Derived  Needs 


■  First  glance  should  tell  overall 
health  and  what’s  up  and  what’s 
down  (i.e.,  core  services) 

■  Want  as  much  information 
presented  as  possible  (i.e., 
aggregation)  on  one  screen 

■  Global  display  that  provides  a 
status  of  alerts  and  sensor  status 

■  Pull  data  and  consolidate  relevant 
facts  into  an  integrated  display 
(i.e.,  provides  summary 
information  by  layer  -  e.g.,  TCP/IP 
model) 

■  Drill  down  for  additional 
information 


MITRE 


Requirements 


Accurate  and  timely,  to  include  IW 
and  general  status  notifications 

Easy  on  the  eyes  (i.e.,  salience) 

Standardized  views 

Easy  to  use  (i.e.,  intuitive  -  not 
much  training  needed,  people  are 
changing  all  the  time) 
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■  Other  Efforts 


Community  Requirements 

■  Primarily 

Situational  Awareness/Common  Operational 
Picture  (COP) 

Dynamic  Performance  Monitoring 

Ability  to  centrally  monitor  and  collect 
configuration  information 

Need  visualization  tools  for  analyst,  operator,  and 
decision  support  and  effective  indications  and 
warnings 

■  Secondarily  -  Related 

Computer  Network  Defense  (CND)  Concept 
Exploration 

Dynamic  Mapping  of  Systems  and  Connections 

Need  to  provide  commander  with  positive  control 
of  information  resources 

Need  to  prioritize  and  dynamically  allocate  access 

Need  ability  to  centrally  manage  security  posture 
of  enterprise  applications  and  equipment 


Requirements 


Present  different  views 
of  managed  objects  on 
the  display  to  meet  the 
need  of  users 

Provide  multiple  level 
drill-down  capabilities 
(additional  linkages  to  be 
established  between 
modules  in  the  future  for 

Provide  the  capability  of 
launching  element 
managers  and  other 
network/system 
managers  for  a  physical 
object 

Present  information 
associated  with  the 
managed  objects 
including  object 
dependency  data 
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Requirements 


Human  Machine  Interface  (HMI)  Factors 


■  User  Interface:  Specification  of  a 
conversation  between  the  user 
and  the  computer 

-  GUI 

-  Display 
Zones 

-  Messaging 

Dialogue  Tone  and  Terminology 
Function  Keys 

-  Color  Selection 

■  Reporting:  Major  output 
functionality,  accuracy  and 
clarity  are  fundamental,  focus  on 
formatting  and  generation 


System  Functionality:  Applicability  of 
functions,  data  capture  and  format, 
processing  capabilities,  and  system 
support  to  retain  usefulness 

-  Usability 

-  Processing 
System  Errors 

-  Data 

-  Audio  Outputs 
Maintenance 

Support  Documentation  &  Training: 
Proper  and  effective  use,  complete 
routine  and  unique  tasks,  and 
troubleshoot  problems 

-  System  Documentation 
Training  and  Training  Material 


7 


MITRE 


©  2006  The  MITRE  Corporation.  All  rights  reserved 


HMI  Considerations 


Requirements 


■  User  should  expect: 

To  be  aware  of  what  to  do  next 

-  To  know  system  expectation 

Data  checking--has  been  entered 
correctly  or  not 

-  Explanation  of  delays 

-  Task  completion  notification 

Standardized  formatting,  information, 
instructions 

Messages  appear  in  the  same  general 
area  and  remain  long  enough  to  ‘read’ 
them 

Dialogue  be  limited  to  one  idea  per 
frame,  whether  paging  or  scrolling 
through  the  zone 


■  Criteria: 

Ease  of  use 

Human  Factors 
Engineering 

Documentation 

Training 

Security 

Maintainability 

-  Reliability 

Interoperability 
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Operational  Platform  Requirements 


Requirements 


■  Best  viewed  on  a  large  screen  display 

(Note:  plasma  screens  have  burn  in  issues) 

■  High-end  graphics  card: 

-  ATI  Radeon 

-  Visiontek  NVIDIA  GeForce 
Appian  Jeronimo  Graphics  Card 


■  Personal  Computer  (PC) 
based  systems 

Desktops 

(e.g.,  Dell  Dimension) 
Laptops 

(e.g.,  Dell  Inspiron) 

■  Windows  Operating 
Systems  (OSs) 


XP  (Experience) 
Windows  2000 
NT  (New  Technology) 


MITRE 
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Systems  Design  and  Development 


Design  & 
Development 


■  Identify  related  components  and  needed  integration 

-  Conflicts 

-  Similarities 

■  Breakdown  the  problem  into  workable  modules 

-  Alert  Visualization  (AlertViz)  provides  global  alert  visualization 

Exploration  Visualization  (ExViz)  provides  the  same  data  with  queried 
data  for  specific  time  frames 

Connection  Visualization  (ConnViz)  provides  connection  log 
visualization  in  a  grid  sorted  by  Source  (Src)  or  Destination  (Dst) 
Internet  Protocol  (IP)  on  one  axis,  time  on  a  second  axis,  and  quantity 
of  connections  in  the  third  axis 

■  Prioritize  development  focus  based  on  requirements 

■  Design  to  Concept  -  Provide  a  viable  3D-display  prototype  to  meet 
network  operational  needs 

■  Interactive  development  -  Frequent  updates  -  Quick  turn  around 
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Integration 


Design  & 
Development 


IWViz 

*  3D  Visualization 

*  Situational  Awareness 


IWQuery 


Parts  & 
People 
Working 
,s  A  Whole 


Sensors 

*  Intrusion  Detection 
Sensors 

*  Generates  alerts 


Interface  between 
system  components 
Query  capability 


Director 
>  Stores  alert  information 
Provides  detailed 
alert  information 


AlertViz 


Design  & 
Development 


Represent  Alerts:  ■  Alerts  filtered  on: 

Cubes  over  designated  sites  -  String  Match  Events 

-  Information  Window  containing  relevant  (e-9->  BAD_PASSWORDS) 

Events  (e.g.,  Distributed  Port  Probe) 

-  Protocols  (e.g.,  Transmission  Control 
Protocol  [TCP]) 

Source  or  Destination  IP 

Area  of  Responsibility  (AOR) 

-  Sensor 


alert  data 

Date  and  time  of  last  alerts  received  and 
destination  location  indicator 

Usability: 

Configurable  files  for  tailoring 

■  Filter  designators  and  descriptors 

■  Coordinates  (e.g.,  site  latitude/longitude) 

■  Sensor  collectors 

■  Hot  IP  list  with  blinking  option 

-  User  selectable  color  coding 
Point  and  click/auto  data  fill 
User  defined  profiles 

-  Visual  sizing 


General 
Information  Window  Area  of 

Responsibility 
Toolkit  Page 


Wed  May  02  20:12:45  2001  GMT 
c Event:  RPC  High  Port  Activity 

□  SrcIP:  204.208.27.216 
cDstIP:  140.140.126.42 

□Service:  [TCP] 

□  Status:  FR 

□Sensor:  shania  AOR:  AFMC 
Matches:  *9  RTA:  146 


Events 
Toolkit  Page 


Filter  IPs  and 
CDS  IP  Search 
Toolkit  Page 


Last  Alert 

Information  Window 


MITRE 


Last  alert:  Thu  Hay  03  00:59:34  2001 
Timeframe:  4.99  hrs  Alert  count:  3339 

Hot  IP  Enabled 
Closed  Connection 
Distributed  Port  Probe 

RPC  High  Port  Activity 


Item  Session 


Am  1 

Area _ef  flesf oofl 
0  ICC 
0,  AE1C 
El  AFCSB.T 
Eiafwc 
e  mt 
0  AFSPC 
B  AHG 

B  CESTAJ 
0  MM 
E  mn 


fiSKUIL 


Am 

Eieati 

L!  25F  Hitlxt  Jeuoi 

E  S  Distinct  Events 

E!  I  Umiti 
0  JPACHEjEHlTflJLD 
; :  Already  a  Lacked 
0  Ancoaly 
E  Sad  Fragnent 
E  Sad  II  fljtian 
E  Dad  ftp  idditss 
0  Sad  ftp  port 
0  Sleek  Successful 
0  Bogus  Fragaeat 


Am 

Filter  I>s 

Source  IP 


i  k  i  i 


Destination  IP 


(?)  Inclusive  OErchsive 


Accept 


EL 
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ExViz 


■  Includes  all  functional  capabilities 
included  on  the  AlertViz  module 
as  well  as: 

Information  Window  containing 
bytes  sent  and  received 

Date  and  time  of  last  alerts 
received  and  destination  location 
indicator 

■  Query  capability  on: 

Source  IP/Destination  IP 

-  Date 
Time  range 
Status 

-  Bytes  transferred 

-  Destination  Port  # 


MITRE 


Design  & 
Development 


ExViz 


Last  alert:  Thu  Jan  10  09:59:59  2002 
Timeframe:  0.92  hrs  Alert  count:  93 


Thu  Jan  10  09:34:47  2002  GMT 
□Event:  Windows  Registry 

□SrcIP:  206.38.145.21 
□DstIP:  137.244.190.200 
□Service:  cnipfTCP] 


ExViz 

Zero  Bytes 

□  Include  zero  bytes  sent 

□  Include  zero  bytes  reed 


wdows  Registry 


□Status:  PR 


iSensor:  alanis  AOR:  APMC 
ByteSent:  1786  ByteRecd:  1372 
Matches:  1  RTA:  13 


Thu  Jan  10  09:27:14  2002  GMT 
i  Event:  NTSETFAIL 

□SrcIP:  131.25.206.98 
□DstIP:  128.159.101.4 

□Service:  netbios-ssn[TCP] 

□  Status:  FR 

□Sensor:  monkees  AOR:  AFSPC 
ByteSent:  492  ByteRecd:  132 
Matches:  1  RTA:  2 


Site: 

Kunsan 

Sensor: 

supremes 


.  Bagram 
Kandahar  - 

Ca4i%lf^r4kBRfMi  Allrapco<^abad  - 

Dha*fMc.ttWMMRCENJ) 

Bahra" 

ARCENT  580  . 


kaiwviz 


■  Alerts  filtered  on: 


Capabilities  included  on  the  AlertViz 
module 

-  Zero  Bytes  Filtering 
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Fielding  and  Deployment  Activities 


Fielding 

& 

Deployment 


■  Pre-Deployment  Activities 
(Preparation) 

Coordination  activities 

Build  contact  list  and 
locations 

Interim  Certificate  to 
Operate  (ICtO) 
approval 

Prepare  platform  and 
support  documents 

Survey  requesting 

configuration 

information 

Baseline  software 

Coordinate  travel  with 
Points  of  Contact 
(POCs) 

Accreditation 


Deployment 
Activities  (Fielding) 

General  briefing 

Check  setup  and 
configuration  (e.g., 
Director  connection) 

-  Test  run  setup 

Train  target  users 

Review  System 
Change  Request 
(SCR)  process 

Question  and  Answer 
(Q&A)/collect  initial 
comments 

-  Out  brief 


Post-Deployment  Activities 
(Sustainment) 

Update  deployment 
process 

Prepare  after  action  report 

Follow-up  on  submitted 
enhancements 

Send  out  software  and 
documentation  updates  to 
deployed  locations 

Review  CND  visualization 
needs  and  support  options 

Provide  maintenance  and 
help  desk  support 

Finalize  documentation  - 
worked  throughout 
process 
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Deployment  Rationale 


Fielding 

& 

Deployment 


■  IWViz  offers  capabilities  not  available  in  Commercial-Off-The-Shelf 
(COTS)  software 

■  No  licensing  fees 

■  Satisfies  Intrusion  Detection  System  (IDS)  related  visualization 
requirements 

■  Conditions — Provides  prototypes  to  users  “as  is”  with  ongoing 
support  from  government  sponsor  until  successful  sustainment 
transition  occurs,  currently: 

-  Working  assessment  of  network  impact  or  interoperability — 

None  expected  or  experienced  in  the  laboratory  operation 

User’s  manual  provided — 

Includes  setup  directions 

-  Training  provided — 

Initially  at  deployment,  On-The-Job,  Call  in  technical  support 

Initial  load  and  configuration — 

Users  to  support  updates  and  new  version  installs 
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Technology  Transfer 

■  IWViz  moved  from  the 
prototype  stage  and  was  ready 
to  migrate  to  an  operational 
technology 

MITRE  not  chartered  or  staffed 
to  provide  product  sustainment 

-  Necessary  to  implement  an 
IWViz  technology  transfer  plan 

■  Concerns 

-  Intellectual  Property 

-  Conflict  of  Interest 

-  Legal  &  Ethical  (Contact) 

-  Lose  of  Direction 

■  Select  best  way  forward 

Which  is  the  easiest  and  least 
confining? 

Which  ones  do  we  want  to 
pursue,  could  be  one  or  many? 


MITRE 


Technology 

Transfer 

Options 

Commercial  License  -  Nonexclusive: 

Can  require  them  to  give  back  copies 

■  We  transfer  the  software  directly  to 
selected  companies 

■  Sponsor  transfers  the  software  to 
contractors  and/or  companies 

-  Open  Source 

■  Need  Public  Release 

■  Open  to  all  folks  national  and  abroad 

■  Hard  to  track  who  has  it 

-  Cooperative  Research  And 
Development  Agreement  (CRADA)  - 
Joint  government/industry  R&D 
partnerships  which  share  resources 

Industry  Standards  -  Influence 

External  Publications  -  Public  domain 
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Growing  Interest  in  Technology  Transfer 


Technology 

Transfer 


■  Venture  investors  looking  for  solid  technology 
underpinnings  in  investment  opportunities — reaction  to  dot 
com  era  mistakes 

■  Research  institutions  increasingly  willing  to  partner  with 
venture  community  for  additional  revenue  opportunities 

■  Homeland  Security  and  other  government  initiatives  driving 
interest  in  potential  applications  of  advanced  technologies 
being  developed  in  research  institutions 

■  Mid-Atlantic  technology  commercialization  organizations 
ideally  positioned  at  the  crossroads  of  industry  and 
government  technology  hubs 

■  Tech  transfer  still  relatively  new  (Bayh-Dole  Act  passed  in 
1980) 
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Last  alert:  Ihu  Hay  03  00:59:34  2001 
Timeframe:  4.99  hrs  Alert  count;  3339 
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Site: 
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RPC  High  Port  Activity 


Barksdale  ATE 
Beale  afb 
Cannon  API 
Davis  Monthan  AFB 
Dyess  APB 
Ellsworth  AFB 
Ho llonan  AFB 
Langley  AFB 
Minot  afb 
Moody  AFB 
Mountain  Home  AFB 
Mils  AFB 
Off  lit  fc  AFB 
Seymour  Johnson  AFB 
Shaw  AFB 
wAitenan  afb 


HT  Server  Get  Info 

Stem  Session 


^Deselec^^ 


!i.r 


Alert  Visualization  (AlertViz) 


*®coo<y»e  ^Oo 


Exploration  Visualization  (EXViz) 


Connection  Visualization  (ConnViz) 


Real  time  situational  awareness  allowing  for  quicker  response  in  decision  making 
Allows  for  initial  situational  assessment  and  predictive  support 


Primary  Support  Team 


■  Therese  Metcalf — Technical  Lead-MITRE 

■  Ken  Beyer — Government  Project  Lead 

■  Dr.  Mike  Wingfield — Lead  Developer-MITRE 

■  Tim  Farias — Testing  &  Technical  Support-MITRE 

■  Interface  Developer — Varied  Government  Support 

■  Users — Government  Personnel 
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IgGRAl] 


Enterprise-Wide 
Systems  Engine) 


the  Government 
and  the  Globe 


Spanni 


Assistant  Secretary  of  Defense 

for  Networks  and  Information  Integration/ 

DoD  Chief  Information  Officer 


Mr.  Mike  Kern 
Deputy  to  the  ASD(NII)/DoD  CIO 
Enterprise  Wide  System  Engineering 


WA 


GIG  EW  SE  Mission 

Ensure  the  GIG  meets  the  End-to-End 
interoperability  and  performance  requirements 
needed  to  support  DoD  business,  intelligence, 
enterprise  information  environment  and  war 
fighting  operations. 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 


Global  Information  Grid  EW  SE  Environment 
spans  across... 

•  The  entire  Federal  Government 

•  Multiple  programs  and  statutory  authorities 

•  Many  technical  domains 


DoD  Mission  Area  (MA)  Portfolios 


Core 


Tactical 
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Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 
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GIG  EW  SE  Challenge  -  Ensure  technical  integrity, 
interoperability,  and  performance  across  the  GIG 


Packet  Level  End-to-End  (E2E)  Models 


Broad  range  of  users  consistent 
with  1-4-2-1  Ops  scenario 


Legacy  and  future  services 
Clear  definitions  of  service  functions 


Significant  E2E  Performance  Reduction 

Summary  Comparison 

2MB  File;  Ingress  Link  Speed=lMbps  (25%  load) 


Satellite 

/  LX*  - 

\j\  Current  &  Legacy  transport 


Current  &  Legacy  transport 
Broad  range  of  end-to-end  paths 
100+  use  cases  defines 


1,200 


1,000 


l  Standard  TCP 
I  Large  windows  TCP 
I  Enhanced  TCP 


6x  increase  in  E2E 
transfer  time 


Wide  Range  of  E2E  Solutions 


Revise  use 
case 

Implement  requirements  4  ^ 
common  ^  ■  W  planning  and 

provisioning 

E2E 

solution 


standards 
and  protocol: 


ition 

Adjust 

priorities  and  Alter  routing 

QoS  strategy  strategy 

Change  GIG 
component 
requirements 


<<  200 


Little  motivation  for  segments  to 
implement  solutions 

/ ■ 

Terrestrial 


High  Loss 
Low  Latency 

Wireless 


Low  Loss 
High  Latency 

Satellite 


High  Loss 
High  Latency 

End-to-End 


Individual  network  l  program  solutions 
DO  NOT  ensure  E2E  performance 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 


EW  SE  Approach  -  Lead  a  community  effort 
to  enable  a  portfolio  decision  process  to: 


►  Provide  EW  SE  services  needed 
to  oversee  GIG  evolution 

►  Maintain  a  GIG  enterprise-wide 
coherent  technical  and 
operational  baseline  to  insure 
interoperability 

►  Establish  GIG  enterprise-wide 

analysis  capabilities  for  E2E 
Performance  Assessment 

►  Oversee  GIG  enterprise-wide 

experiments  &  federated 
emulation/test 

►  Establish  a  GIG  compliance 
management  program 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 
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Maintain  a  GIG  EW  coherent  operational  and 
technical  baseline 


Align  documentation  framework  for  Program  Managers 

•  GIG  description  documentation  is  unclear 

•  Test  plans  difficult  to  develop 

•  Weak  linkage  to  operational  needs 

•  Business  and  Intelligence  mission  areas 
not  currently  included 

•  -7000  pages  of  conflicting  guidance 


CONOPS 


Architecture 


ICDs 


NCE  (and  Other)  JFCs 
NCOE  (and  Other)  JICS 


GIG  NetOps 
CONOPS 

Dev. 

Coord. 


FTE=? 

FTE=? 


Emerging 


GIG  NetOps 
Architecture 


Dev.  FTE=? 
Coord.  FTE=? 


Dev.  FTE=? 

Coord.  FTE=? 

IA  Architecture 

Dev.  FTE=? 
Coord.  FTE=? 


Joint  GIG  “Requirements” 


GIG  NetOps 
Architecture 
Version  2.0 

Dev.  FTE=? 
Coord.  FTE=? 


Dev.  FTE=? 

Coprd.  FTE=?~§ 


GIG  Mission  Area  (MA)  ICD 


Dev. 

Coord. 


FTE=? 

FTE=? 


Others? 


Transformational 
Communications 
Architecture  2.0 

Dev.  FTE=? 
Coord.  FTE=? 


o  a 

LU  LU 

o  o 
a  o 


LU  < 

y>  o 


Net-Centric  Implementation  Document  (NCID) 


Key  Interface  Profiles  (KIP)  Dev  FTE=? 

Coord.  FTE=? 

1 


Dev.  FTE=? 
Coord.  FTE=? 


Dev.  FTE=? 
Coord.  FTE=? 


►  Program  Direction 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government  ? 

and  the  Globe  0 


New  Coherent  operational/technical  baseline  improves 
stability  and  raises  confidence  in  guidance 


Revised  framework  will  provide  structure  and 
traceability  similar  to  that  of  a  document  tree 


and  the  Globe  |  ' 


New  Baseline  feeds  directly  into  established 
oversight  and  decision  processes 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 
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End-to-End  Performance  -  why  is  it  so  complicated? 


GIG  performance  questions  across  the  community 


►  End-User 

•  Which  applications  can  I  successfully  use  over  the  GIG? 

•  What  performance  can  I  expect  on  GIG? 

•  Which  technologies  should  I  employ? 

•  What  are  the  impacts  of  different  operating  conditions? 

►  Transport  Developer 

•  What  service  and  applications  must  I  support? 

•  Is  the  E2E  performance  acceptable? 

•  What  are  inter-segment  interface  impacts? 

•  What  are  the  E2E  impacts  of  segment  performance  trades? 

►  Application  &  Services  Developer 

•  How  will  my  application  perform  across  the  GIG? 

•  Is  the  E2E  performance  acceptable? 

•  What  are  acceptable  service  architectures? 

•  What  are  the  E2E  impacts  of  protocol  and  messaging  trades? 

►  Operator 

•  Where  should  servers  be  deployed? 

•  What  are  acceptable  operating  loads? 

•  What  SLA  performance  do  I  need? 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 
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Performance  Assessment  Framework  highlights  critical 
problems  to  guide  end-to-end  trades  and  implementation 


Composite  Network  Performance 


Application  &  Service  Messaging 


Access 

Links 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 
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Global  Information  Grid  analysis,  emulation  and  test 
in  the  acquisition  lifecycle 


Summary  -  EW  SE  provides  the  Technical  Framework 
for  DOD’s  GIG  Business  Process  Improvement 


DoDD  5144.1  provides  DoD  CIO 
authority  and  responsibilities  for 
defining  and  implementing  GIG 


GIG  Policy  must  support 
existing  decision  processes 
(Requirements  Generation, 
Acquisition,  Budget  and 
Operations)  beyond-GIG 


4^  GIG  Policy  establishes  authority 
for  EW  SE  processes  and 
GIG  Technical  Baseline 
development  and  establishes 
authority  for  GIG  program 
compliance  with  Technical 
Baseline 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Performance 

Assessment 


DoDD  5144.1  - 
ASD(NII)/DoD 
CIO 


Operations 

Resource  Mngt  Policy 

Policy 

DoDD  7045.14 

CJCSM  6231. xx 

(PPBE) 

Acquisition  Policy 
DoDD  5000 
(DAB/MDA) 


Requirements 

Policy 

CJCSI  3170.01 
t.TCTDSt 


O  Implementing 
Processes 


Policy 

Framework 
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For  further  information 


•  Dr.  Tony  DeSimone 

•  Tonv.Desimone@osd.mil 

•  (703)  607-0344 

•  Mr.  Ed  Tavares 

•  Edward.Tavares.ctr@osd.mil 

•  (703)  607-0356 

•  ASD(NII)  DoD  CIO  Web  Page 

•  www.dod.mil/cio-nii 


Global  Information  Grid 

Enterprise-Wide  Systems  Engineering  (GIG  EW  SE) 


Spanning  the  Government 
and  the  Globe 
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Dr.  Daniel  Leshem 

Corporate  Chief  Scientist 

Rafael  Armament  Development  Authority  Ltd., 

Haifa,  Israel 
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New  Perceptions  in  Coping  with  SoS 

i  SoS  Definitions 

>  OSoS  Characterizations,  Parameters  and  Types 

■  Rationale  for  Defense  Industry  Entry  in  OSoS 
Development 

OSoSE  Versus  Systems  Engineering 

■  OSoSE  Recommendations 
OSoS  Management  Highlights 
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How  to  Define  SoS? 

Family  of  Systems?  Enterprise  SoS?  Complex 
System?  or  Super  System? 

■  For  example:  one  fighter  aircraft  incorporates 
many  systems,  but  SoS  refers  to  entire 
squadron! 

i  To  be  clear,  I  propose  the  term 

Operational  SoS  (OSoS ) 
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Operational  SoS  Definition 


Ensemble  of  platforms,  armaments, 
eqn|}m^,l^C‘f|  personnel,  fjalrtfjg  and 
maintenance  facilities. 

These  synergistically  work  together  to 
achieve  set  of  tasks  within  a  defined 
mission,  according  to  operational 
methodologies  driven  by  combatdoctrine 
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OSoS  Example 
FCS  (Future  Combat  System) 

OSoS  includes 


Platforms 

Weapons 

C4I2SR 

Doctrine 

Training  facilities 
Fighting  Units 
Soldiers 
Maintenance 
ILS 

(Integrated  Logistic  Su 


Materiel 


Leader  Development 


Training 


Objective  Force 


i.  hits  of  Employment 


T*-., 


installations 


- 


Units  of  Action 

.  #■'  J&k  if1* 

I  CS  Burnt/ ion 


oldiers 


i  J 


|p*> 

Doctrine 


NDIA  SE  October  2006 


New  Perceptions  in  Coping  with  SoS  -  D.  Leshem  Rafael  Proprietary 


5 


FCS  OSoS  -  Concept  of  Operations 


Joint  ISR 


Objective 
Operational  & 
Tactical  Forces 


NLQST  > 

Com  man' 


CONUS  Higher  HQs 
En  Rotate  Planning 
Rehearsal 


Joint  & 
Coalition 

Assets 


V  Satellite  Communications 

^ 


Legend:  Interfaces 
Intra-FCS 
Interoperable 


National  ISR 


FCS  Battle  Brigade 
C2  Subsystem 


Dismounted 
Combat  Team  A 


Combat  Team  B 


Coalition  Forces 
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OSoS  Characterizations 

Trained  personnel  are  integral  part  of  OSoS 

OSoS  building  blocks  can  be  distributed  at 
remote  locations  and  interconnected  by 
communication  networks 

C4I  systems  are  critical  “glue”  for  combining 
separate  systems  into  OSoS  bHB 

Coordinated  operation  of  all  OSoS  building 
blocks  is  key  factor  in  effective  and  efficient 
mission  accomplishment 

OSoS  concept,  architecture  and  capabilities  are 
being  consolidated  in  parallel  with  evolutionary 
development  of  its  operational  methodology 
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OSoS  Types 

■  Dedicated  vs  Virtual  OSoS 


Preplanned  vs  gradually-developed 
OSoS 
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Rationale  for  Defense  Industry  Entry 
into  OSoS  Development 

i  Defense  market  customers  new  attitude  is  to  deal 
with  OSoS  -  defining  requirements,  evaluating 
and  ordering  OSoS  -  and  not  separate  systems 

i  Many  advantages  for  main  contractor 

Increasing  reputation  as  leading  defense  industry 
Enabling  direct  interaction  with  customers  including 
End  Users 

Improving  opportunity  to  incorporate  own  advanced 
technologies 

Improving  own  capabilities  when  coping  with  OSoS 
Systems  Engineering  (OSoSE) 
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OSoSE  vs  Systems  Engineering 

Starting  OSoS  Program 

Requirements  consolidation  and  management 

Functional  Analysis 

Interfaces  definition 

Modeling  and  Simulation 

Defining  and  executing  V&V 

Evolutionary  behavior  of  OSoS 

Architectural  analysis  of  OSoS 

Deep  involvement  of  interested  parties 
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OSoSE  vs  Systems  Engineering 


Starting  OSoS  Program 

Reauiremontc  rrmcrJirlatirm 


ament 


4-  Very  long  processes  together  with 
interested  parties  and  customers  to  refine 
concept 

4-  Early  consolidation  of  Strategic  Partners 
i-  Considerations  of  using  state-of-the-art  or 
just  Foreseen  Technologies 
4-  Starting  interim  developments  to  reduce 
risks  and/or  to  verify  new  concept  (usually 
the  proven  concepts  become  constraint  in 
ihe  full  scale  program) 
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OSoSE  vs  Systems  Engineering 

Starting  OSoS  Program 

Requirements  consolidation  and  management 


Very  high  level  of  complexity  in  consolidating 
requirements  for  new  OSoS: 

•  many  customers  with  different  interests, 

•  evolutionary  nature  of  development,  and 

•  necessity  for  interoperability 

Requirements  develop  and  are  updated  during 
entire  OSoS  life  cycle 

Connectivity  and  interfaces  with  neighboring  OSoS 
under  development 

Advanced  software  tool  is  needed  to  handle  the 
huge  requirements  scope  (the  original  +  derived 
ones)  and  their  complex  linkages 
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OSoSE  vs  Systems  Engineering 

Starting  an  OSoS  Program 
Requirements  consolidation  and  management 
Functional  Analysis 
Interfaces  defi 
Modelin 
Defir 
Evol 
Arch 
Deep 


4-  Processes,  tasks  and  scenarios  in  OSoS 
are  complex,  involving  many  components 
4-  These  create  interactive  constraints  and 
mutual  effects 

4-  Conclusion:  Methodical  Functional 
Analysis  is  critical  for  architectural 
consolidation  of  OSoS 
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OSoSE  vs  System 


r 


Starting  OSoS  Progrq 
Requirements  consq 
Functional  Analysis 
Interfaces  definition 
Modeling  &  Simulatio 
Defining  and  executii 
Evolutionary  behavio 
Architectural  Analys 
Deep  involvement  of 


4-  Systems  Engineering 
defines  mechanical, 
electrical,  computational 
-  connectivity  and  MMI 
interfaces 

4-  OSoS  provides  upper 
layer  of  different  types  of 
scenarios  and  tasks: 
Physical:  different 
configurations  and 
Logical:  different  tasks 

4-  Result:  Different 
interface  requirements 
for  various  applications 
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OSoSE  vs  Systems  Engineering 


Starting  an  OSoS  Progr, 
Requirements  consolid 
Functional  Analysis 
Interfaces  definition 
Modeling  &  Simulation 
Defining  and  executi 
Evolutionary  behavior 
Architectural  Analysis 
Deep  involvement  of  inte 


4- Not  possible  to  produce 
complete  model  of 
OSoS  to  simulate  all  its 
functionality  as: 

&  Infinite  cases  of 
possible  events  and 
scenarios  included 
Very  difficult  to 
simulate  human 
behavior 
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OSoSE  vs  Systems  Engineering 


Starting  OSoS  Program 
Requirements  consolid 
Functional  Analysis 
Interfaces  definition 
Modeling  &  Simulation' 
Defining  and  executing 
Evolutionary  behavior  c 
Architectural  Analysis 
Deep  involvement  of  ini 


4-  These  situations 
require: 

#  Using  many  models  to 
cover  all  hierarchy 
levels 

#  Simulating  tasks 
management  with 
Humans-in-the-Loop 

#  Developing  special 
simulation  concepts 
and  tools  to  cope  with 
OSoS 
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OSoSE  vs  Systems  Engineering 

Starti 
Requ 
Func 

Interf  _ 

Modeling  &  Simulation 
Defining  and  executing 
Evolutionary  behavior  of  OSoS 
Architectural  Analysis  of  OSoS 
Deep  involvement  of  interested  parties 


i-  Complex  V&V  procedures:  analysis, 
simulations,  integrations  and  tests 
4-  Verification  of  the  design  in  stages, 
combined  with  risk  analysis  and  plans 
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OSoSE  vs  Systems  Engineering 


>4-  Complex  V&V  procedures:  analysis, 
simulations,  integrations  and  tests 
>4-  Verification  of  the  design  in  stages, 
combined  with  risk  analysis  and  plans 


Modeling  &  Simulation 
Defining  and  executing  V&V 


Evolufi 
Archil 
Deep 
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4-  Interfaces  verification  at  all  levels, 
scenarios  and  tasks 
4-  OSoS  validation  in  series  of  high-level 
preplanned  test  scenarios 
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vs  Systems  Engineering 


i-  OSoS  Life  Cycle  Concept:  taking  into  account 
existing  systems,  future  development  and  stages 
in  life  cycle  and  life  time 

i-  Adaptive  development:  considering  unexpected 
changes  in  directions  as  result  of  operational 
lessens  learned  by  early  fielded  models 
i  Development  environments:  as  teams, 
subcontractors,  knowledge,  experts,  tools  - 
are  chanaina! 


Evolutionary  behavior  of  OSoi 
Architectural  Analysis  of  OSoS 
Deep  involvement  of  interested  parties 
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OSoSE  vs  Systems  Engineering 


4-  Definition:  Model  description  (physical, 
functional,  operational,  software  etc)  at  OSoS 
upper  level  in  its  relevant  environment,  considering 
its  evolutionary  and  open  system  natures 
4-  Architectural  Analysis  of  OSoS  require  very  high 
level  of  professionalism  in  following  issues: 

►  Methodology 

►  Tools 

►  Training 

►  Experts 


Architectural  Analysis  of  OSoS 


Deep  involvement  of  interested  parties 
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OSoSE  vs  Systems  Engineering 


±-  Range  of  interested  parties  usually  with 
contradictory  interests!  -  including  main 
contractor  and  subcontractors,  customers 
taking  part  in  development  and  end  users 

4-  OSoS  military  aspects  require  close 
involvement  of  customer  as  early  as 
possible  with  professional  teams  who  can 
make  difficult  decisions! 

i-  Close  cooperation  with  customer  and 
mutual  obligations  are  essential  for 
successful  building  of  OSoS! 


Architectura 


naiysis  o 

Deep  involvement  of  interested  parties 
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OSoSE  Recommendations 

>  Methodology 

>  New  special  upper  level  of  Systems 
Engineering  Process  to  be  defined  (V  model 
recommended) 

>  Interoperability  and  MOSA  (Modular  Open 
System  Approach)  in  OSoS  to  be  adopted 

>  Professionalism  of  OSoS  Architecture  and 
function  of  Chief  Architect  (compared  to 
Chief  Systems  Engineer)  to  be  cultivated 
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OSoSE  Recommendations 

>  Organization 

>  Chief  Architect,  responsible  for  all  aspects 
of  OSoS  Engineering,  directly  subordinate 
to  Program  Manager/Management 

>  Working  with  Integrated  SE  Team  (ISET) 
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Integrated  SE  Team  for  OSoS 

ISET 
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OSoSE  Recommendations 


>  Tools 

>  Central  database 

>  Creating  and  preserving  logical  connections 
with  full  traceability 

>  On-line  coordination  between  all  development 
bodies 

>  Automatic  documents  production 

>  Fast  transformation  from  UC  analysis  to 
working  software  implementation 

>  Massive  supportability 

>  Examples:  Core,  Requisite  Pro,  Clear  Case, 
Rhapsody,  Doors 
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OSoSE  Recommendations 

>  Critical  technology  disciplines  and 
infrastructure  required  for  OSoSE 

>  Information  (collecting,  processing,  distribution) 

>  Intelligence  (collecting,  decoding,  data  fusion, 
processing,  distribution) 

>  Sensing  (electro-  optical,  acoustics,  RF,...) 

>  C4I2SR 

>  Interdisciplinary  and  multidisciplinary  know-how 
such  as: 

>  Image  processing  with  navigation 

>  Communication  with  missile  guidance,  control  and 
navigation 

>  Intelligence  with  signal  processing,  pattern  recognition; 

>  and  more... 
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OSoS  Managements  Highlights 


i  Coping  with  OSoS  provides  main 
contractor  with  many  advantages  and 
benefits  -  and  it’s  worth  it! 

i  Spiral  development  of  OSoS  ensures  use 
of  most  updated  technologies  and 
capabilities  available  while  dealing  with 
enemy  emergent  threats,  in  process  that 
allows  step-by-step  fielding 
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OSoS  Management  Highlights 


i  Recommended  to  manage  OSoS 
architecture  and  systems  engineering 
processes  through  Chief  Architect  and 
Integrated  SE  Team 

i  Use  specific  gates  and  checklists  as 
necessary  management  tool  in  building 
OSoS 

■  Common  methodologies,  tools  and 
Infrastructure  for  all  parties  and  bodies 
involved  in  development  of  OSoS  is  key  to 
success! 
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New  Perceptions  in  Coping  with  OSoS 

Summary 

*!  OSoS  (Operational  SoS)  is  proposed  as 

common  term  to  which  Defense  industry  can 
refer 

4!  New  level  of  complexity  -  not  just  “more” 
regular  Systems  Engineering  -  it  requires 
development  of  new  concepts,  approach  and 
methodologies 

4i  It’s  a  challenge! 

4i  It’s  worth  it! 
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New  Perceptions  in  Coping  with 

System  of  Systems 


Dr.  Daniel  Leshem 

Corporate  Chief  Scientist 

Rafael  -  Armament  Development  Authority  Ltd.,  Israel 

dleshem@rafael.co.il 
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i  C4I2SR 

-Command  Control,  Communication, 
Computers 

-  Intelligence,  Information 

-  Surveillance 

-  Reconnaissance 
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Managing  the  'Trick-Bag" 
of  I  ntersystem  Coupling 

J  effrey  S.  Levin 
Presented  to: 

NDI A  9th-Annual  Systems  Engineering 
Symposium,  San  Diego,  CA 

2  5- October- 2006 
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"Law  of  Strong  Connections 


n 


“A  system  is  a  collection  of  parts,  no  one  of 

which  can  be  changed.” 


“In  systems,  all  other  things  are  rarely  equal.” 


Weinberg,  Gerald  M.  (2001).  General  Systems  Thinking  (Silver  Anniversary  Edition).  New 
York:  Dorset  House  Publishing,  p.  162. 
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1 1  Ml  1 

Baa 

Coupling  versus  Autonomy 

(OSD  AT&L,  Air  Warfare  (December  2002).  "Unmanned  Aerial  Vehicles  Roadmap,  2002-2027,"  p.  41) 

Autonomous  Control  Level  a 


FIGURE  4.4-1 :  ALTCNC^OUS  CONTROL  LEVEL  TREND. 


Closed  systems  are  artificial  creations  of  scientists  and  engineers 
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Managing  I  ntersystem  Coupling 


Propositions 


■  PI:  A  modicum  of 
coupling  achieves  best 
results 


A  General  Model  of  Coupling 


-  Either  end  of  the  coupling 
spectrum  should  be  avoided 

■  P2:  View  Coupling  as  the 
effect,  not  just  a  cause 

-  The  operational  situation 
should  dictate  Coupling 
Required,  not  vice  versa 
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Coupling:  Types  &  Measures 
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1)  I  nter-Capabilitv: 


2)  Complex-  Dynamic: 


3)  I  nter- Nodal 


C  =  5.0 

^Index  =  0.00 

N  *  K  =  50 
E /  Emax  =  1.00 


C  =  5.0 

^Index  =  0.00 

N  *  K  =  25 
E/  E  =  0.50 

'  max 


C  =  6.7 

^Index  — 

N*  K  = 

^mav 


0.40 

20 

0.40 


max 

C  =  7.2 

^Index  =  0.44 
N  *  K  =  20 
E/  Emax  =  0.40 

C  =  8.0 

^Index  =  0-42 

N  *  K  =  20 
E/  Emax  =  0.40 


"Law  of  Mass  Action"  or  'The  System  Concept" 
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Notional  Problem  Application: 
Standoff  Attack  of  Moving  Targets 


Launch 
PI  atform 


Weap  on 
instantaneous 
Footprint 


Targding 

Sensor 


Subsonic 

Weapon 


True 
Position  at 


Target 
Position  at 
7 - 


PROBLEM:  “Target  Location  Error”  (TLE)  is  dynamic 
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Actor  ->  Factor  Linkages 

CAPABILITY 

FACTORS  ACTORS  SYSTEMS 
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NETWORK 
CONFI GS. 


WEAPON 

SEARCH 


SENSOR_A 


SENSOR  B 


LAUNCH  / 
PLATFORM 


WEAPON 
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Minor  Axis 
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Sensor  Precision 


■  Values:  Low  Precision  Vs  High  Precision 


All  else  being  equal,  high  sensor  precision  decreases  AOU  size 
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Targeting  Latency 

■  Values:  High  (min)  Vs  Low  (sec) 


All  else  being  equal,  low  latency  decreases  AOU  size 
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||BBa 

Sensor  Bilateration  Geometry 


■  Values:  Low  Azimuth  Vs  High  Azimuth 


Aircraft  2 


Aircraft  1 


All  else  being  equal,  high-azimuth  geometry  decreases  AOU  size 
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Sensor  Data  Fusion 


■  Values:  Not  Available  Vs  Available 


All  else  being  equal,  sensor  data  fusion  decreases  AOU  size 
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||B8a 

Weapon  Search 


■  Values:  Wide_  Open  Vs  Reduced 


All  else  being  equal,  reduced  search  decreases  the 
probability  of  detecting  a  neutral  vessel 
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ipESa 

Data  from  Design  of  Experiment 
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SeekerSear 
ch (Reduced 
Vs  Open) 

SensorPr 
ecision 
(High  Vs 
Low) 

Fusion 
(Yes  Vs 
No) 

Latency 
(Low  Vs 
High) 

BilatGeo 
metry  (Hi- 
Az  Vs  Low- 
Ax) 

Yla: 

Max_Allow_ 
Density 
(PartialWid 
e-Open  Skr) 

1 

Reduced 

High 

Yes 

Low 

Hi-Az 

213,800 

2 

Reduced 

High 

Yes 

Low 

Lo-Az 

40,100 

3 

Reduced 

High 

Yes 

High 

Hi-Az 

247 

4 

Reduced 

High 

Yes 

High 

Lo-Az 

247 

5 

Reduced 

High 

No 

Low 

5,940 

6 

Reduced 

High 

No 

Low 

5,940 

7 

Reduced 

High 

No 

High  | 

230 

8 

Reduced 

High 

No 

High  | 

230 

9 

Reduced 

Low 

Yes 

Low 

Hi-Az 

7,517 

10 

Reduced 

Low 

Yes 

Low 

Lo-Az 

548 

11 

Reduced 

Low 

Yes 

High 

Hi-Az 

197 

12 

Reduced 

Low 

Yes 

High 

Lo-Az 

107 

13 

Reduced 

Low 

No 

Low 

700 

14 

Reduced 

Low 

No 

Low 

700 

15 

Reduced 

Low 

No 

High 

91 

16 

Reduced 

Low 

No 

High  | 

91 

17 

Open 

High 

Yes 

Low 

Hi-Az 

5,365 

18 

Open 

High 

Yes 

Low 

Lo-Az 

1,006 

19 

Open 

High 

Yes 

High 

Hi-Az 

179 

20 

Open 

High 

Yes 

High 

Lo-Az 

164 

21 

Open 

High 

No 

Low 

298 

22 

Open 

High 

No 

Low 

298 

23 

Open 

High 

No 

High 

164 

24 

Open 

High 

No 

High  | 

164 

25 

Open 

Low 

Yes 

Low 

Hi-Az 

94 

26 

Open 

Low 

Yes 

Low 

Lo-Az 

94 

27 

Open 

Low 

Yes 

High 

Hi-Az 

164 

28 

Open 

Low 

Yes 

High 

Lo-Az 

89 

29 

Open 

Low 

No 

Low 

94 

30 

Open 

Low 

No 

Low 

94 

31 

Open 

Low 

No 

High  | 

76 

32 

Open 

Low 

No 

High  | 

1 

76 

Tightly  Coupled  Condition 


Ql:  Which  factor  is 
most  important? 

Q2:  Which  factor 
combination  is  most 
important? 


Loosely  Coupled  Condition 
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Data  Analysis  -> 

Benefit  of  Tight  Coupling 


CAUTION:  Tight  coupling  will  be  hard  to  achieve  operationally  &  programmatically 
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HKL 


Data  Analysis  -> 
Requirement  for  Coupling 


(9  0.80 

£ 

LU 

UJ  0.70 

•a 

o> 

.h  0.60 

3 

cr 

<D 

q;  0.50 


3 

O 

O  0.30 


Vessel  Density  Versus  Factor  Coupling  Required 


Blue  Water 
Regions 


i 


r 

+- 


Operational  Condition:  Estimated  Average  Vessel  Density 


Littoral 

Regions 

l 

30%  coupling  covers  100%  of  all  operational  conditions  studied 


Only  an  extreme  condition  requires  tight  coupling 


dPL 
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Data  Analysis  -> 

Performance  Sensitivity  to  Coupling 


NOTE:  In  some  cases,  the  same  value  of  coupling 
results  in  high  variation  in  performance  WHY? 


25  October  2006 


Clearly  it  matters  which  capability  factors  are  coupled  together 
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aata 


Data  Analysis 
Capability  I  nteraction 


NOTE:  Latency  is  the  most  important  driver 
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3-Way 

Interaction 
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r  li  ii 

lam 

Basic  Network  Configurations 
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•  3rd  Question:  I  n  general  which  network  configuration  works  best? 


I .  Fully  Connected  1 1 .  Circle 


III.  Chain 


IV.  Y- Branch 


V.  Snokes 


Tightly  Coupled 
Decentralized 


System  Type  Versus  Coupling 


A  system’s  organizational  structure  influences  its  performance  KHI 
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|U  J  I  l 

■5ml 
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NOTE:  Tightly  coupled  structure  has  lowest  performance 
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System  Structure  Vs  Performance 


NOTE:  Tightly  coupled  structure  has  lowest  performance 


dPL 
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Coupling  Vs  Organizational  Efficiency 

(Caroll  &  Burton,  2000) 


Coupling  Versus  Performance  {1 /Duration,  #of  Days) 


Q0D  020  040  050  000  1  0D  120 


Coupling  Index  (K  t  K-max} 


Coupling  Versus  Performance  (1/Project  Errors) 

Performance 

- - - - - *  "  ’ 

_ 

0 

so  ft  a  ft 40  o.eo  o.eo  uuo  ta 

Coupling  Index  {K  l  K-max) 

Coupling  Versus  Performance  (1/  Cost) 

to 

u 

C 

ts 

o 

r 

p 

£L 

# - 4 - + 

^  |  M  

— p— fj=-1 

— N  6 

N=5 

N  10 

o.aa  uji)  dw  o.bu  a.  to  i.uu  i  jj 

Coupling  Index  |K  /  K-max) 


NOTE:  Best  performance  occurs  when  coupling  =  0.30-0.50  (~Chain  &  Y_Branch) 


slide  21 


Managing  I  ntersystem  Coupling 


25  October  2006 


Coupling  Vs  Communication  Efficiency 


NOTE:  As  coupling  increases,  COMMS  efficiency  decreases  exponentially 
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Conclusions  on  Tight  Coupling  ->  One  size 
does  not  fit  all 


Can  exponentially  increase  your  capability 
Hard  to  achieve  operationally  &  programmaticall 
Requires  holistic  acquisition  strategy  a/a 
Results  in  decreased  COMMS  efficiency  &  organizational  performanc 
Risky  ->  "Normal  Accidents"  &  single  points  of  failure 
Need  to  view  entire  system  holistically^] 

It's  hard  to  achieve  holistic  view  of  entire  system 
Tight  coupling  may  not  be  an  operational  requirement 


Moderate  coupling  may  be  best  way  to  manage  environmental  uncertainty? 


L 


II 
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Unstable  Environment/ 
Complex  Situation 
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Further  Research 

Basis:  Daft  (1998),  Perrow  (1984) ,  Weick  (1976) 


Stable  Environment/ 
Simple  Situation 


Ideal 

Region? 


Coupling 


PROBLEM:  Socio-technical  Systems  =  Leadership  +  Organizational  +  Technology 
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Coupling  ~  Circular  Causality? 
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QUESTI ONS? 


J  effrey  S.  Levin 

J  ohns  Hopkins  University  Applied  Physics  Lab 
Aviation  Systems  Engineering  Group 
I  effrev.Levin@ihuapl.edu 

240-228-3533 
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Overview 


■  Strategic  Interests 

■  Life  Cycle  SE 

■  SE  in  Pre-Acquisition  (Prior  to  Milestone  /  Key 
Decision  Point  A) 

■  Early  SE  Pilot 

■  SE  for  Systems  of  Systems  (SoS) 

■  ROE  and  Perspectives 

■  Enabling  Capabilities 

■  Cases  and  Issues 

■  Focus  Areas  for  SE  Planning  and  Measuring 

■  SE  Perspectives 

■  AF  SE  Vision 
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Strategic  Interests 


■  Implement  a  robust  engineering  vision  across 
the  life  cycle  in  all  four  Air  Force  product  lines 
(air,  space,  weapons,  command  and  control) 

■  Institutionalize  the  role  of  Chief  Engineer  as 
the  senior  technical  advisor  supporting  the  Air 
Force  Acquisition  Executive 

■  Grow  and  mentor  the  next  generation  of  Air 
Force  technical  leaders 
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Life  Cycle  SE 


■  “ Life  Cycle  SE”  encompasses  the  entire  set  of 
scientific,  technical,  and  managerial  efforts 
needed  to  plan,  develop,  acquire,  integrate, 
verify,  field,  operate,  maintain,  improve,  and 
sustain  a  system  to  provide  a  needed  capability 

■  Core  SE  Technical  Processes  ■  Specs  and  Standards 


■  Technical  Management  Processes  ■  Program  Protection 


■  System-of-Systems  (SoS) 

■  System  Safety  and  ESOH 

■  Software 

■  Human  Systems  Integration 


■  Manufacturing  Readiness 

■  Maintenance  Engineering 

■  Integrity  Programs 

■  ...  numerous  others  ... 


OUTCOME:  Mission  Assurance  --  Operational  Safety ,  Suitability ,  and 
Effectiveness  (OSS&E)  -  Throughout  the  Product  /  System  Life  Cycle 
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DEMIL  &  DISPOSAL 


SE  in  Pre-Acquisition 
Disciplined  Application  Required 
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SE  in  Pre-Acquisition 


■  What  it  is: 

■  The  tie  between  JCIDS  and  the  Analysis  of  Alternatives 
(AoA) 

■  A  disciplined  process  to  scope  capability  needs  and 
develop  concepts 

■  The  process  required  to  do  necessary  groundwork  for 
a  successful  AoA 

■  A  means  to  identify  candidate  technologies  and  assess 
their  TRLs 

■  Good  SE 

■  What  it  is  not: 

■  An  actual  AoA 

■  "Gaming  the  system"  to  favor  a  solution 
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SE  in  Pre-Acquisition 

Early  Documentation 

■  Standard  methodology  --  “Analysis  of  Problem” 
as  precursor  to  formal  Analysis  of  Alternatives 

■  Describes  how  SE  processes  translate  capability 
statements  into  families  of  concept  designs/approaches 

>  T rade  study  process 

>  Key  ground  rules/constraints 

>  Decision  criteria 

>  Methodology  for  populating  knowledge  base 

■  Describes  how  operational  context  (architectures  and 
military  utility)  drives  these  translations 

■  Basis  for  Technology  Development  Strategy 

■  TDS  should  make  up  ~75%  of  content  of  SEP  submitted 
at  Milestone  /  Key  Decision  Point  A  for  selected  concept 
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SE  in  Pre-Acquisition 

Example 


■  Warfighter  needs  the  capability  to  cross  a  body  of  water 

■  Initial  pass  through  this  process  yields  various  methods 


■  Airlift 

■  Bridge 

■  Catapult 

■  Drive  around 


■  Drive  through 

■  Ferry 

■  Tunnel 

■  etc. 


■  Further  analyses  offer  parametric  trades  within  a  method 
(e.g.,  bridge),  considering  depth,  width,  current,  etc. 

■  "If  you  build  400  yards  upstream  where  the  channel  is  narrower, 
you  will  only  need  3  support  pilings  instead  of  4  ...” 

■  "If  you  build  1000  yards  downstream  where  the  current  is  slower, 
you’ll  need  5  pilings  and  20%  more  material  for  the  road,  but  you 
can  finish  10%  sooner  and  the  span  can  take  15%  more  live  load  ...” 

■  Applying  this  process,  the  Acquirer  will  NOT  determine 
what  type  of  bridge  is  best  -  that  comes  out  of  the  AoA 
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Early  SE  Pilot 


■  SMC  effort  began  June  06 


■  SMC/XD  to  select  candidate  need  from  AFSPC 
Mission  Needs  Statement  library 

■  Selected  needs  will  be  used  to  validate  entire 
process  by  developing  at  least  two  concepts 

■  Deliverables  include  details  of  each  step  in  process 
diagram,  guide  to  implementation  criteria,  and  draft 
SEP  for  each  concept 

■  Project  will  expand  to  AFMC  Product  Centers 
in  FY07 

■  Best  Practices  and  Lessons  Learned  will  shape 
policy 
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Early  SE  Pilot 

Concept  Development  Process  Diagram 


INPUT  -  CAPABILITY  STATEMENT 
(SOURCE  /  FORMAT  NOT  SPECIFIED) 


|  REQUIREMENTS  ANAL  YSIS 


VALIDATE  /  REFINE 
I  CAPABILITY  STATEMENT 


DEFINE  OPERATIONAL 
CONTEXT,  AFFECTED 
MISSION  AREAS,  ETC. 


DEVELOP  MOEs  FOR  INITIAL 
MILITARY  UTILITY  ASSESSMENT 


VERIFICATION 


DEFINE  APPROPRIATE 
RANGES  OF  MILITARY 

UTILITY 

REFINE  OPERATIONAL  CONTEXT,  MOEs, 
AND  RANGES  OF  MILITARY  UTILITY 


FUNCTIONAL 
ANALYSIS 


IDENTIFY  AND  DOCUMENT  KEY 
CONSTRAINTS,  GROUNDRULES, 
AND  ASSUMPTIONS;  INCLUDE  MOPs 


REVIEW  AND  REFINE 
CONSTRAINTS  /  GROUNDRULES  / 
ASSUMPTIONS  /  MOPs 


VERIFICATION 


REFINE  TRADE  PARAMETERS 
AS  NEEDED 


Develop  general  classes  of  concepts  /  approaches 

Identify  tradeable  parameters  and  fixed  values  to  facilitate  comparison  of  concepts 
Identify  relevant  technologies,  test  approaches,  and  required  demonstration  resources 
Conduct  preliminary  technical  and  economic  analysis  to  filter  concepts 
Conduct  parametric  analyses  to  establish  ranges  for  key  factors 
Identify  models,  processes,  tools,  etc.  to  be  used  in  detail  trades;  identify  needed 
modifications  /  changes  for  future  use 

>  Operational  scenarios  >  LCC  Model 

>  Simulations  >  Other 

Document  all  decisions,  analysis  results,  process  deviations,  etc. 


*  -  CAN  BE 
INDEPENDENT 
OUTSIDE 
ORGANIZATION 


SYNTHESIS 


Integrity  -  Service  -  Excellence 


11 


SE  in  Pre-Acquisition 

Expectations 


■  Disciplined  application  of  Pre-A  SE  will: 

■  Flow  operational  needs  through  concepts  into  programs 

■  Integrate  the  “illities”  up  front  into  concept  definition 

■  Build  a  technical  knowledge  base  that  migrates  with 
concepts  to  programs 

■  Policy  for  Pre-A  SE  will: 

■  Drive  linkage  of  concepts  to  operational  architectures 
(Air  Force,  Coalition,  and  Joint) 

■  Facilitate  better  decision-making  at  MS/KDP  A  and  B 

■  Policy  for  Pre-A  SE  will  not: 

■  Provide  guidance  idea  generation 

■  Direct  conduct  of  AoA  or  EoA  studies 

■  Guide  overall  capabilities  integration  activities 
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SE  in  Pre-Acquisition 

Summary 

■  Aim  to  institutionalize  disciplined  and  consistent 
SE  application  throughout  the  life  cycle,  across 
all  AF  product  lines  (air,  space,  weapons,  C2) 

■  MUST  start  in  the  earliest  stages  of  concept 
development,  BEFORE  formal  program  initiation 

■  Early  SE  is  an  Investment  to  reduce  risk  in  later 
program  phases 

■  Pilot  project  will  define  rigorous  Early  SE  process 

■  Policy  will  follow  validation  in  FY07 _ 

ULTIMATE  RESULTS 

>  Better  technical  planning,  better  integrated 

>  More  confidence  in  programs  entering  acquisition 
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SE  for  Systems  of  Systems  (SoS) 

Rules  of  Engagement 

■  Systems  of  Systems  (SoS)  result  when  independent  and 
useful  systems  are  integrated  into  a  larger  system  that 
delivers  unique  capabilities 

■  Both  the  SoS  and  the  constituent  systems  consist  of  parts, 
relationships,  and  a  whole  that  is  greater  than  the  sum  of  the 
parts 

■  While  the  SoS  is  a  system,  all  systems  are  not  SoS 

■  SE  for  SoS  deals  with  planning,  analyzing,  organizing, 
and  integrating  the  capabilities  of  a  mix  of  existing  and 
new  systems  into  a  SoS  capability  greater  than  the  sum 
of  the  capabilities  of  the  constituent  parts 

(Defense  Acquisition  Guidebook,  Chapter  4) 

■  Keys:  Definitions,  Development,  Acquisition,  Operations 

Guide  to  Systems  of  Systems  (SoS)  Engineering:  Considerations  for  Systems  Engineering 
in  a  SoS  Environment  --  draft  OUSD  (AT&L)  publication;  anticipated  release  Dec  2006 
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Perspectives  - 1 


SE  for  SoS 


■  Definitions 

■  System  of  Systems 

■  Family  of  Systems 

■  Federation  of  Systems 

■  Architecture 

■  Enterprise 

■  Others  ?? 

■  Development 

■  Holistic  view 

■  Aggregation  of  platform-level  efforts 

>  Focus  on  physical  interfaces  and  functional  /  information 
exchanges 

>  Degree  of  integration  often  vague 
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Perspectives  -  2 


SE  for  SoS 


■  Acquisition 

■  DoD  processes  largely  specific  to  each  Service/Agency 

■  Systems  acquired  independently,  even  to  satisfy  similar  sets  of 
requirements 

■  Contractor  development  processes  differ  greatly,  even  for  similar 
systems  in  similar  product  areas 

■  Further  cultural  differences  in  AF 

>  Space  systems  <  »  Non-space  systems 

>  Weapon  systems  «  1  Business  and  IT  systems 

>  Product  Centers  '  >  Logistics  Centers  '  >  Test  Centers 

■  Operations  /  Employment 

■  Often  used  in  new  combinations 

>  Near-infinite  number  of  subsets  of  constituent  elements/systems,  etc. 

>  Near-infinite  number  of  dynamic  configurations 

■  Often  used  in  new  environments  and  operational  scenarios 

■  Often  used  with  new  supporting  cast 
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SE  for  SoS 

Enabling  Capabilities  -  Example 


Capability:  Targeting  of  Coordinate-Seeking  Weapon  (CSW) 

( all-weather ,  air-to-ground ,  GPS/INS-guided  munitions) 


SoS  Architecture 

^ 

Acquirer 

Passer 

Processor 

Passer 

Platform 

Weapon 

Representative  Constituent  Systems 


Sniper 

Link  16 

AOC 

Link  16 

F-22A 

GBU-38  (JDAM) 

JSTARS 

DCGS 

AWACS 

B-52 

GBU-39  (SDB) 

Predator 

GCS 

F-15E 

BAO/TAC 

AWACS 

F-16C/D  Blk  50 

None  of  these  systems  was  designed  with  CSW  targeting  in  mind ,  and 
only  a  few  of  the  systems  were  designed  to  interface  with  each  other 
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SoS  Enabling  Capabilities 
Defined  by  Architectural  Views 


Technical  View  (TV)  System  View  (SV)  Operational  View  (OV) 


Boundary 


Modules  Interfaces 


Adapted  from  Open  Systems  Joint  Task  Force 


Interactions 


Boundaries 


How  the  contractor  builds  it 
How  we  support  &  maintain  it 

<  What  we  buy  ► 

TV  "Success  Criteria  " 

All  system/subsystem  components  functior 
properly;  robust  designs  use  “plug-&-play” 
open  interfaces  and  industry  standards 

SV  "Success  Criteria " 

All  robust  platforms/systems  operate 
safely  &  function  properly  to  deliver 
discrete  capability  in  the  battlespace 

Where  &  how  we  use  it,  and 
assess  value  &  effectiveness 

OV  "Success  Criteria " 

All  assets  safely  interoperate  as  SoS; 
integrated  capability  delivery  in  the 
battlespace  is  essentially  “plug-&-fight” 
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SE  for  SoS 

Cases  &  Issues  - 1 

■  New  developments 

■  Seen  as  the  exception  rather  than  the  rule 

■  Affordability  constraints 

■  Centralized  funding  and  management 

■  Role  of  integrator  vs.  role(s)  of  developer(s) 

■  Integrated  legacy  systems  for  new  capability 

■  Will  probably  be  the  most  common  approach 

■  Defined  in  interoperability  (“plug-and-fight”)  context 

■  Discovery  of  interactions,  especially  in  ad  hoc 
configurations 

■  Inconsistent  and  disparate  management  of 
configurations,  data,  etc. 

■  Test  /  M&S:  formal  test,  verification  and  validation, 
experimentation 
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SE  for  SoS 
Cases  &  Issues  -  2 


■  Collaboratively  developed  constituent  systems 

■  Defined  in  terms  of  a  common  architecture  to  guide 
development  of  new  systems  and  platforms 

■  Greater  number  of  stakeholders,  with  both  parochial 
and  holistic  interests 

■  Data  sharing  among  multiple  contractors 

■  Definition  and  management  of  common  architecture(s) 

■  Sustainment  of  existing  systems 

■  Different  pace  of  updates  for  different  systems  at 
different  points  in  their  life  cycle 

■  Increase  capabilities  by  technology  refresh  /  insertion 

■  Often  with  different  levels  of  documentation  and  data 

■  Diminishing  (and  often  volatile)  sources  of  support 

■  Local  management  /  funding  (particularly  true  for 
business  and  IT  systems) 
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SE  for  SoS 

Summary 

■  Driven  by  fiscal  and  operational  realities 

■  Fundamental  SoSE  processes  (technical  &  technical 
management)  largely  the  same  as  for  “classical”  SE 

■  Greater  emphasis  on  architecting  and  interfaces 

■  Integration  challenges:  test  &  real-world  environments 

■  Defining  architectures  to  link  systems  and  platforms 

■  Experimentation  as  a  development  tool 

■  Managing  utilization  of  assets  acquired  and  operated  under 
disparate  systems  and  policies 

■  Unique  management  and  governance  issues 

ULTIMATE  RESULTS 

>  Robust,  responsive  capabilities  delivery;  better  integrated 

>  More  confidence  in  system  performance 
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SE  Perspectives 

Acquisition,  Operations,  Integration,  Architecture 


Capability 

Planning 


Concept 

Refinement 


Technology 

Development 


System  Development 
&  Demonstration 


Production  &  Deployment 
Operations  &  Support 
Disposal 


Level  2: 

Hardware  /  Software  Building  Block 


Supplier/  OEM 

Supply  Chain  Mgmt 


Level  1 : 

Hardware  /  Software  w 


3 
£ 


Views  of  the  “universe' 

Acquisition  Operational 


Test  &  integration  focus  (notional) 

DT&E  M&S  /  Experimentation  ffi™ 


OT&E 


Architecture  views 

(spans  are  not  authoritative) 
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SE  Perspectives 

Acquisition,  Operations,  Integration,  Architecture 


Project  Engineers 
(Program  &  Contractor) 

Logistics  Centers 

Supplier/  OEM 

Supply  Chain  Mgmt 


Capability 
Planning  y 


Concept 

Refinement 


Technology 

Development 


System  Development 
L  &  Demonstration  j 


Level  3: 

rrF® 

Functional  Area  ( 

\Y  3 - o  1 

0  c/ 

(e.g.,  Integrated  Core  Processing) 

Level  2: 

Hardware  /  Software  Building  Block 

Level  1: 

Hardware  /  Software  Component 

Production  &  Deployment 
k  Operations  &  Support 
Disposal 


3 
£ 


Views  of  the  “universe' 

Acquisition  Operational 


Test  &  integration  focus  (notional) 

DT&E  M&S  /  Experimentation  ffi™ 


OT&E 


Architecture  views 

(spans  are  not  authoritative) 
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SE  Perspectives 

Acquisition,  Operations,  Integration,  Architecture 


Project  Engineers 
(Program  &  Contractor) 

Logistics  Centers 

Supplier/  OEM 

Supply  Chain  Mgmt 

Views  of  the  “universe 

Acquisition  Operational 


Capability 

Planning 


System  Development 
&  Demonstration 


Production  &  Deployment 
Operations  &  Support 
Disposal 


Level  3^ 

Functional  Area 

(e.g.,  Integrated  Core  Processing) 
Level  2: 

Hardware  /  Software  Building  Block 
Level  1 : 

Hardware  /  Software  Component 


Test  &  integration  focus  (notional) 

DT&E  3  tic 


OT&E 


Architecture  views 

{ spans  are  not  authoritative) 
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SE  Perspectives 

Acquisition,  Operations,  Integration,  Architecture 


Weapon  System  CEs 
&  Tech  Staff 

Operators  & 
Maintainers 

Project  Engineers 
(Program  &  Contractor) 

Logistics  Centers 

Supplier/  OEM 

Supply  Chain  Mgmt 


Views  of  the  “universe 

Acquisition  Operational 


Capability 

Planning 


System  Development 
&  Demonstration 


A 


Production  &  Deployment 
Operations  &  Support 
Disposal 


Level  4: 

Major  Subsystem 
(e.g.,  Avionics  Suite) 

Level  3: 

Functional  Area  . 

(e.g.,  Integrated  Core  Processing) 

Level  2: 

Hardware  /  Software  Building  Block 
Level  1 : 

Hardware  /  Software  Component 


Test  &  integration  focus  (notional) 

DT&E  3t  tic  §8 


OT&E 


Architecture  views 

(spans  are  not  authoritative) 
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SE  Perspectives 

Acquisition,  Operations,  Integration,  Architecture 


Platform  AWeaponlfiR 
. . (e.g.,JSF) . 

Level  4: 

Major  Subsystem 
(e.g.,  Avionics  Suite) 


Stores  Stations 


Capability 

Planning 


Technology 

Development 


SAF/AQ,  SAF/US, 
PEOs 

MAJCOMs 

Weapon  System  CEs 
&  Tech  Staff 

Operators  & 
Maintainers 

Project  Engineers 
(Program  &  Contractor) 

Logistics  Centers 

Supplier/  OEM 

Supply  Chain  Mgmt 


Concept 

Refinement 


System  Development 
&  Demonstration 


Production  &  Deployment 
Operations  &  Support 
Disposal 


Level  3: 

Functional  Area 

(e.g.,  Integrated  Core  Processing) 
Level  2: 

Hardware  /  Software  Building  Block 


Level  1 : 


Hardware  / 


* 


Views  of  the  “universe"  Test  &  integration  focus  (notional)  Architecture  views 

Acquisition  Operational  DT&E  111111 07&E  (spans  are  not  authoritative) 
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SE  Perspectives 

Acquisition ,  Operations ,  Integration ,  Architecture 


OUSD(AT&L)  /  JCS 
COCOMs 


SAF/AQ,  SAF/US, 
PEOs 

MAJCOMs 

Weapon  System  CEs 
&  Tech  Staff 

Operators  & 
Maintainers 

Project  Engineers 
(Program  &  Contractor) 

Logistics  Centers 

Supplier/  OEM 

Supply  Chain  Mgmt 


Capability 

Planning 


Technology 

Development 


A 


System  Development 
&  Demonstration 


A 


Production  &  Deployment 
Operations  &  Support 
Disposal 


Level  6: 

Force  Structure  / 
Syste  m  -of-Sy  ste  ms 


Level  5: 


Platform  /  Weaponll^fitatei 
(e.g.,JSF)  ■ 


Level  4: 

Major  Subsystem 
(e.g..  Avionics  Suite) 


Level  3: 

Functional  Area 

(e.g.,  Integrated  Core  Processing) 


Level  2: 

Hardware  /  Software  Building  Block 
Level  1 : 

Hardware  /  Software  Component 


Views  of  the  “universe' 

Acquisition  Operational 


Test  &  integration  focus  (notional) 

DT&E  atic 


OT&E 


Architecture  views 

(spans  are  not  authoritative) 
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■  Program  Requirements  ■ 

■  Capabilities,  CONOPS,  KPPs 

■  Statutory  /  regulatory 

■  Specified  /  derived  performance 

■  Certifications 

■  Design  considerations 

■  Technical  Staffing  /  Organization  ■ 

■  Technical  authority 

■  Lead  Systems  Engineer 

■  IPT  coordination 

■  IPT  organization 

■  Organizational  depth 

■  Systems  Engineering  Process  ■ 

■  Technical  Processes 

■  Technical  Management  Processes 

■  Process  Improvements 

■  Key  Tools  and  Resources 

■  Trade  Studies 

■  Linkage  to  Contractor  SE  Effort 
*  -  Based  on  OSD  SEP  Preparation  Guide 


Integrity  -  Servi 


Focus  Areas  for 
SE  Planning  * 

Technical  Baseline  Management 

■  Who  is  responsible 

■  Definition  of  baselines 

■  Requirements  traceability 

■  Specification  tree  and  WBS  link 

■  Technical  maturity  and  risk 

Technical  Review  Planning 

■  Event-driven  reviews 

■  Management  of  reviews 

■  Technical  authority  chair 

■  Key  stakeholder  participation 

■  Peer  participation 

Integration  with  Overall 
Management  of  the  Program 

■  Linkage  with  other  program  plans 

■  Program  manager’s  role  in  technical 
reviews 

■  Risk  management  integration 

■  Test  and  logistics  integration 

■  Contracting  considerations 


e  -  Excellence 


28 


PARAMETER  VALUE 


Focus  Areas  for 
Measuring  Progress 


■  Representative  Technical  Performance 
Measures  (TPM)  parameters 

■  Hardware  -  weight,  speed,  power, 
cooling,  cross-section,  bandwidth 

■  Software  -  throughput,  lines  of  code 

■  Verification  -  test  asset  deliveries,  test 
points  completed  with  valid  data 

■  Logistics  -  reliability,  maintainability 

■  Integration  -  physical  and  information 
interface  definitions;  verification  plans 


Earned  Value  Management 
System  (EVMS)  data 

■  Cost  variances 

■  Schedule  variances 

Program  planning 

■  Staffing 

■  Subcontracting 

■  Specification  approvals 

■  Closure  of  review  actions 


Threshold  Objective 
Lower  bound  Upper  bound 

Achieved  to  date  Plan 

TIME 


Monitortrend;  take  action  here 

Plan  is probablyachievable 


Overly  optimistic  "get-  well" plan 
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Focus  Areas  -- 
Emerging 


■  Technical 

■  Manufacturing  Readiness 

■  Human  Systems  Integration  (AF/SG  lead) 

■  Specifications  and  Standards 

■  Governance  &  Oversight 

■  MDA  Certification  (Section  801  of  National  Defense 
Authorization  Act  for  FY06  [Pub.  L.  109-163]) 

■  System  &  Software  Assurance  (Security  &  Program  Protection) 

■  Multi-Faceted 

■  Enterprise-level  SE 

■  Industrial  Base 
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Visions 


1.  Algorithms 


■  Consistent  application  of  rigorous  life  cycle  SE  in  all  AF  product  lines ,  enabled  by 
a  skilled  workforce  and  a  policy  framework  with  an  integrated  life  cycle  perspective 

■  Delivery  and  support  of  quality  systems/SoS/software,  on  cost  and  on  schedule 


jses 
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“We  demand  rigidly  defined  areas 
of  doubt  and  uncertainty!” 


*  * 


*■_ 


*  * 


t  * 
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•  Program  Management  oversees  an  expansive  umbrella  that  carries  a  bundle  of  business  and  technical 
activities.  Within  that  umbrella,  Systems  Engineering  supports  Program  Management  by  integrating  these 
independent  and  dependent  activities  into  one  synergistic  entity  that  will  develop  and  deliver  systems.  Indeed 
Systems  Engineering  is  a  foundation  and  an  enabler  for  Program  Management. 

•  On  a  mature  multi-purpose  airlift  program,  Program  Management  concentrates  on  several  key  program  items, 
such  as  costs,  timely  delivery,  people,  quality,  risks,  and  ultimately  customer  satisfaction  and  confidence.  These 
program  items  are  a  natural  part  of  the  systems  engineering  process.  In  view  of  these  program  items,  Systems 
Engineering  focuses  on  stabilizing  and  enhancing  the  systems  engineering  process. 

•  As  joint  partners,  Program  Management  and  Systems  Engineering  organizations  must  work  together  to  define 
and  understand  their  diverse  objectives.  Convergence  of  fnese  objectives  will  lead  to  best  practices  that 
accomplish  business  goals,  such  as  reducing  cycle  time  without  impacting  quality,  and  deliver  a  product(s)  or 
system(s)  to  the  satisfied  customer. 

•  For  feedback  to  Program  Management,  a  Systems  Engineering  Scorecard  measures  systems  engineering 
performance  across  the  multi-purpose  airlift  program.  Inherently,  this  metric  will  positively  benefit  the  corporate 
enterprise.  Company  financial  results  and  performance  can  meet  or  exceed  shareholders’  expectations 
because  good  application  of  systems  engineering  will  provide  total  cost  visibility  and  will  lead  to  reducing 
system  life-cycle  costs,  including  design  and  development  costs. 


Also  within  this  umbrella,  the  users,  such  as  Project  Integrators  and  Project  Managers,  of  the  systems 
engineering  process  can  then  apply  and  utilize  it  to  manage  product  developments)  that  add  new  capabilities  to 

.  Svst  '  "  '  '  r  ‘‘  ‘  ‘  . 


the  multi-pur 
managemen 
Managemen 


Dose  airlift  system.  Systems  engineers  are  critical  members  of  the  project  team  that  aligns  program 
‘  items  and  objectives  with  product  development.  In  sum,  a  strong  relationship  between  Program 
and  Systems  Engineering  is  essential  to  product  or  system  development. 
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•  The  Partnership  Umbrella 

•  Systems  Engineering  Process  Stabilization  &  Enhancement 

•  Users  of  the  Systems  Engineering  Process  at  Multiple  Organization  Levels 

•  Conclusion 
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The  Partnership  Umbrella:  Program  Management 


Employee  Quality  Analysis  & 
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The  Partnership  Umbrella:  Systems  Engineering 


8  Primary  Life 
Cycle  Functions 
of  Systems 
Engineering 


Development - 


Manufacturing/ 

Production/  Training 

Construction 


A 


f _ Disposal/ 

Retirement 


N' - Support 

Operation 


Deployment 


Systems  Engineering  Management 

-  Foundation  for  Program  Management 

-  Program  Stability  Support 
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Program  Management 


Systems  Engineering 


Global 

Economic 


Customer 
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Share 
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Growth 
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Issues 


Risks 
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Tools 
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Maturity 


Workplace 

Safety 


Revenues 


MIS 


Cost 
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Business 

Growth 


Customer 

.Satisfaction 


Workforce 

Training 


Opportunities 

Leadership 


Earned  Value 
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Product 
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Cost 


Suppliers 


TPMs  IMP/IMS 
Engineering 


Labor 

Relations 
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Satisfaction 


Customer 

Needs/ 


Interfaces 


Requirements 


Quality 
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Trade  Studies 
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Tools 
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The  Partnership  Umbrella:  Interrelationships 


Program 

Management 


Program  Objectives 

\ 

/ 

Program  Management  Plan 

Program  Technical  Requirements 


Systems 


Program  Management  Requirements 


^  I  Engineering  I  S 


v 


System  Specification  Type  A 


Development 
Specification  Type  B 


Systems  Engineering 
Management  Plan 


RM&A  Plan 


ILS  Plan 


JL 

Other  Specifications  & 
Standards 


Software 

Engineering 

Plan 


Source:  Blanchard,  Benjamin  S.  (2004)  Systems  Engineering  Management.  New  Jersey:  John  Wiley  &  Sons 
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•  Systems  Engineering  Supporting  Program  Management 

-  Product  Development 

-  Assessments  &  Best  Practices 

•Contractor  Performance  Assessment  Reporting  (CPAR)  Review 
•TA-314  Project  Performance  Assessment  &  Review 

•  Program  Management  Best  Practices 

•  Engineering  Best  Practices 
•Systems  Engineering  Scorecard  Review 

•  Assessments  &  Best  Practices  provide  total  visibility  on  strengths  and 
weaknesses  in  Systems  Engineering  as  well  as  progress  of  improvement  efforts 
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Contractor  Performance  Assessment  Reporting  (CPAR) 


•  Objectives: 

-  Ensure  that  accurate  data  on  contractor  performance  is  current  and  available  for  use 
in  source  selections 

-  Consistently  provide  quality,  on-time  products  and  services  that  conform  to  contractual 
requirements 

-  Effectively  communicate  contractor  strengths  and  weaknesses  to  source  selection 
officials 

•  Systems  Engineering  Supporting  Program  Management: 

-  Use  Award  Fee  Rating  Criteria 

-  Review  Customer’s  AFAST  Database 

-  Review  Award  Fee  Review  Charts 

-  Review  Project  Integration  Weekly  Reports 

-  IMP/IMS  Performance 
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TA-314  Project  Performance  Assessment  and  Review 


•  Objectives: 

-  To  rate,  assess,  and  report  project  performance  to  management  and  the  customer 

•  Systems  Engineering  Supporting  Program  Management: 

-  Review  Technical  Performance  Measurement  (TPM) 

-  Review  Systems  Engineering  Compliance 

•  Requirements 

•  Risk 

•  Verification 

•  Formal  Review 

•  Critical  Action  ltem(s) 


10/26/2006 
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Program  Management  Best  Practices 


•  Objectives: 

-  To  achieve  successful  program  development,  implementation  and  support  based  on 
an  integrated  set  of  Program  Management  Best  Practices 


•  Systems  Engineering  Supporting  Program  Management: 

-  Review  maturity  level  for  program  execution  &  control 

-  Use  program  execution  &  control  best  practice  criteria 

•  Establishment  of  the  program’s  systems  engineering  methodology 

•  Allocation  and  traceability  of  program  requirements 

•  Verification  of  program  requirements 

•  Program  identification  of  TPMs 
•Allocation  and  reporting  of  TPMs 

Business  Plan 


Supplier 

Integration 


Program  Execution  & 
Control 


Organization 


Risk,  Issue  & 
Opportunities 


Help  Needed  & 
Independent 
Review 


Business 

Proposition 


Program 

Communication 


10/26/2006 
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•  Objectives: 

-  Vision  Support  Plan:  Strengthen  Systems  Engineering 

-  Maintain  the  Capability  Maturity  Model  Integration  (CMMI)  Level  5 

•  Systems  Engineering  Supporting  Program  Management 

-  Develop  Engineering  Best  Practices  Self  Assessment  Plan 

-  Review  overall  attributes  associated  with  each  of  the  Best  Practices 

-  Improve  training  materials 

•  Requirements  Management 

•  Interface  Management 
•Technical  Performance  Measures 

•  Trade  Studies 

•  Verification  &  Validation 

-  Conduct  trainings 


10/26/2006 
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•  Objectives: 

-  To  develop  the  method  and  criteria  for  measuring  the  effectiveness  of  SE  at  a  program 
level 

-  To  define  and  ensure  common  application  of  World-Class  SE  processes  and  tools  to 
meet  customer  expectations 

•  Systems  Engineering  Supporting  Program  Management: 

-  Monthly  assess  program-level  systems  engineering  performance 

•  PMBP,  Engineering  BP,  CPAR,  PBMs,  and  TA-314 

-  Monthly  assess  each  IPT’s  compliance  to  systems  engineering  process 

•  AV/FC/SE  IPT,  Aircraft  System  IPT,  Airframe  IPT,  AV/SS  IPT,  Engineering  Integration  IPT,  and 
NCO  IPT 

-  Generate  Monthly  Systems  Engineering  Rollup  Report 


10/26/2006 
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Users  of  the  SE  Process  at  Multiple  Organization  Levels 


LU 


O 

_l 

< 


LU 

(0 


Enterprise  Level 


Program  Level 


Project  Integration  Level 


Project  Level 


Image  Source:  University  of  Toronto  Magazine 


10/26/2006 
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•  While  the  focus  of  Program  Management  lies  in  the  business  arena  and  Systems 
Engineering  in  technical,  the  two  disciplines  share  many  fundamental  principles, 
elements,  and  activities 

-  Program  performance  and  product  performance  are  inter-dependent  in  many 
parameters 

•  These  parameters  can  be  integrated,  mapped,  aligned,  and  mutually  supported 
to  create  a  comprehensible  and  synergistic  relationship 

-  Clear  visibility  of  overall  key  performance  parameters 

-  Consistency  in  assessment  of  overall  key  performance  parameters 

•  This  synergistic  relationship  is  an  essential  enabler  for  the  effective  management 
of  a  program 

•  Good  applications  of  the  SE  Process  is  seen  at  all  levels  of  an  organization  to 
encourage  consistency  and  alignment  for  product  development 


10/26/2006 
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•  Samuel  Son,  Systems  Engineer 

-  Phone:  (562)  982-2209 

-  Email:  samuel.k.son@boeinq.com 

•  Benjamin  Q.  Luong,  Systems  Engineer 

-  Phone:  (562)  593-4668 

-  Email:  beniamin.q.luonq@boeinq.com 


Thank  you! 


Questions?  We  might  have  answers... 


10/26/2006 
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Quality  /  Systems  Engineering 
Interfaces  to  Enhance 
Program  Success 
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October  23-26,  2006 


I  DA 


Dr.  Jay  Mandelbaum 
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*  How  quality  impacts  technical  risk 

*  Involving  quality  early 

*  Encouraging  a  quality  focus 

*  Incorporating  quality  in  the  systems  engineering 
technical  reviews 

*  Concluding  remarks 


Quality  Matters 


Firestone  tire  story 


V-22  Osprey  story  USS  Thresher  story 


-I*  JL  i 


Flaw  Could  Shorten  Raptors’  Lives 
The  News  Herald,  Panama  City,  Fla.  May  3,  2006 


"Off icials  discovered  that  the  titanium  components  [of  the  forward  boom  frames] 
may  have  been  improperly  treated,  creating  the  possibility  that  the  metal  would 
not  last  as  long  as  it  is  supposed  to.  ...  This  is  not  a  result  of  improper  design, 
but  an  issue  with  one  supplier's  manufacturing  process." 


therefore 
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Industry  Has  Adopted  a  New  Paradigm  for 
_ Quality _ 

Quality  is  no  longer  limited  to  a  centralized  functional 
organization  for: 

-  Auditing  /  compliance  with  standards  (certification) 

-  Implementation  of  a  quality  management  system 

-  Inspections  /  searching  for  defects 

Today,  quality  engineers/SMEs  are  typically  matrixed 
to  a  product  or  project  early  in  development 

-  Valued  member  of  IPTs 

-  Closely  linked  with  a  company’s  process  improvement 
initiatives 

-  Assess  if  processes  are  working  as  designed  using 
objective  criteria 

Process  carelessness  leads  to  technical  risk 


Risks  that  Quality  Identifies  and  Mitigates 


Quality  engineers/SMEs  that  may  lead  to  potential 

identify  situations  risks  to  be  mitigated 


-  Inappropriate  application  of 
technical  processes  and/or 
inadequate  procedures 

-  Ineffective  supplier 
management 

-  Ineffective  customer 
engagement 

-  Unidentified  verification 
technologies 


-  Ill-defined  requirements,  a 
breakdown  in  requirements 
flow  down,  or  uneconomically 
producible  designs 

-  Suppliers  with  inadequate 
capabilities;  decreasing 
leverage  with  subtiers 

-  Dissatisfied  customers 


-  Undetected  product  defects 
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Analyses  Show  Prevalence  of  Process 

Based  Risks 


Design  error 


Process  did  not  exist 
or  was  lacking 
sufficient  detail 


Lack  of 
resources/time 
or  did  not  follow 
process 


Process  existed  but 
performer  was 
unaware  of  process 


N  =  89  Design 
Escapes 


Summary  Root  Causes 

*  Lack  of  clear  standard  design 
process 

■  Insufficient  process 
compliance  accountability  and 
enforcement  requirement 

■  Ineffective  design  reviews 


Presented  by  Lisa  Kohl 

Vice  President,  Northrop  Grumman  Space  Technology 
Lean  Aerospace  Initiative  Plenary  Conference 
23-24  March  2004 
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Why  are  these  Risks  Important? 


*  If  not  managed  and  mitigated,  these  risks  may  start 
a  chain  of  events  resulting  in  undesirable  outcomes 

-  Product  defects  discovered  in  production  or  testing 
lead  to  expensive  and  time-consuming  rework 

-  Product  may  not  meet  customer  needs 

-  Product  failures  discovered  in  the  field  lead  to 
degraded  mission  effectiveness  or  mishaps 

*  The  later  these  risks  are  identified,  the  greater  the 
cost  of  corrective  action  and  the  greater  the  delays 
in  schedule 

Early  identification,  management,  and  mitigation  of  important 
process-based  risks  to  a  program  leads  to  less  expensive  and  less 
disruptive  preventive  actions  that  break  the  chain  of  events 
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Best  Practices  for  Reducing  Process-Related 

Risk  to  a  Program 


*  Early  Quality  involvement  with  the  product  line  to 
ensure  that  processes  are  implemented  correctly 

*  Create  conditions  that  encourage  the  entire 
workforce  to  focus  on  quality  along  with  cost, 
schedule,  and  performance 

-  Clear,  unambiguous  and  visible  commitment  from 
management 

*  Develop  strong  linkages  between  the  quality 
engineers/SMEs  on  the  product  or  project  and  the 
SE  technical  reviews 

DoD  should  encourage  industry  to  apply  these  best  practices  and 
follow  them  consistent  with  its  oversight  responsibilities 


Outline 


*  How  quality  impacts  technical  risk 

*  Involving  quality  early 

*  Encouraging  a  quality  focus 

*  Incorporating  quality  in  the  systems  engineering 
technical  reviews 

*  Concluding  remarks 
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Early  Quality  Involvement  (slide  i  of 3) 


*  A  “process  approach”  is  the  identification, 
interaction,  application,  and  management  of  a 
system  of  processes  over  the  life  cycle 

*  Quality  engineers/SMEs  involvement  means 
implementing  processes  correctly  in  order  to 

-  Understand  and  meet  requirements 

-  Evaluate  processes  in  terms  of  value  added 

-  Obtain  results  of  process  performance 

-  Improve  performance  based  on  objective 
measurement 


Don’t  wait  for  production  before  engaging  Quality; 
Don’t  delay  interactions  until  there  is  a  problem 
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Early  Quality  Involvement  <Siide2of3) 

ISO’s  Model  of  a  Process  Based  QMS 


Early  systematic  use  of  a  process  based  quality  management 

system  (QMS)  improves  outcomes 
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Early  Quality  Involvement  <Siide3of3) 

A  Process  Based  QMS’s  Interactions  with  a  Program 


resporsib  tiiy 


inagement 


Meas-JiermsTil 
anay&s  arc! 

jntxov&n'iseni 


Resource 

management 


Customer  Satisfaction 

•  Customer  Reporting 

•  Training  System 

•  Publications 

•  Post  delivery  support 


Customers 


systems 
Engineering 
Prod  Reqts  Alloc 
Allocation, 
verification  & 
validation  plan 
Configuration, 
data,  risk  mgmt 


Software 

Engineering 

•  Design 

•  Integration 

•  Code 

•  Testing 


Customer  Requirements 

•  Product  requirements 
documentation 

•  Program  cost  and  schedule 
performance  data 

•  Formal  customer 
communication 


Management  Responsibility 

•  Management  Review 

•  Quality  policy 

•  Customer  focus 

•  Risk,  issue  &  opportunity  mgmt 

•  Program  Independent  Analysis 

•  Continuous  process  improvement 


Manufacturing 

Product 

•  Fabrication 

•  Assembly 


Integrated  Product 
Definition 

•  Design 

•  Specialty 
Engineering 

•  Support 
Functions 


Product  Support 

•  Performance 
based  logistics 

•  Field  support 
requirements 


Test  Engineering 

•  T est  reqts 

•  Test  planning 

•  Test  conduct 

•  Reports 


Supplier  Management 
and  Procurement 

•  Purchasing  process 

•  Purchase  order  reqts 

•  Purchase  process 
verification 


Manufacturing 

Engineering 

•  Manufacturing 
and  inspection 
work  instructions 

•  Producibility 

•  Workflow 


Measurement, 
Analysis,  and 
Improvement 

•  Metrics 

•  Corrective  &1 
preventive  acti 

•  Continuous  Pr 
Improvement 


Output 


Resource  M: 

•  People 
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•  Training 
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Outline 


*  How  quality  impacts  technical  risk 

*  Involving  quality  early 

*  Encouraging  a  quality  focus 

*  Incorporating  quality  in  the  systems  engineering 
technical  reviews 

*  Concluding  remarks 
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Encourage  Quality  Focus  <siideiof3) 


Values  are  ideas  or  things  accepted  as  good,  desirable, 

and  important  as  codified  by  policy  and  guidance 

-  Example:  “nothing  goes  out  the  door  if  it’s  not  done  right” 

Beliefs  are  what  people  really  think 

-  Example:  people  do  not  ask  penetrating  questions  in 
meetings  to  avoid  elevating  issues 

Behaviors  are  what  the  workers  really  do 

-  Example:  due  to  schedule  or  cost  pressures,  workers  draw 
conclusions  from  incomplete  or  inadequate  data  thereby 
accepting,  ignoring,  underestimating,  or  hiding  risks 


Avoid  mismatches  among  values,  beliefs  and  behaviors 


Adapted  from  David  Leetsma,  Manager  Advance  Planning  Office,  Johnson  Space  Center, 
as  presented  to  the  2006  Conference  on  Quality  in  the  Space  and  Defense  Industries  15 


Encourage  Quality  Focus  (slide  2  <*3) 

Emphasis  in  the  ISO  Context 


Customer  Requirements 

•  Product  requirements 
documentation 

•  Program  cost  and  schedule 
performance  data 

•  Formal  customer 
communication 


Management  Responsibility 

•  Management  Review 

•  Quality  policy 

•  Customer  focus 

•  Risk,  issue  &  opportunity  mgmt 

•  Program  Independent  Analysis 

•  Continuous  process  improvement 


Customer  Satisfaction 

•  Customer  Reporting 

•  Training  System 

•  Publications 

•  Post  delivery  support 


Customers 


N'-anagement 
resporsib  lily 


Resource  Management 

•  People 

•  Competence 

•  Training 

•  Environment 

•  Infrastructure 


Customers 


Measjfenfi&nfi. 
ana  y&s  and 


Measurement, 
Analysis,  and 
Improvement 

•  Metrics 

•  Corrective  & 
preventive  action 

•  Continuous  Prod 
Improvement 


Systems 

Engineering 

•  Prod  Reqts  Alloc 

•  Allocation, 
verification  & 
validation  plan 

•  Configuration, 
data,  risk  mgmt 


Software 

Engineering 

•  Design 

•  Integration 

•  Code 

•  Testing 


Integrated  Product 
Definition 

•  Design 

•  Specialty 
Engineering 

•  Support 
Functions 


Product  Support 

•  Performance 
based  logistics 

•  Field  support 
requirements 


Test  Engineering 

•  T est  reqts 

•  Test  planning 

•  Test  conduct 

•  Reports 


Supplier  Management 
and  Procurement 

•  Purchasing  process 

•  Purchase  order  reqts 

•  Purchase  process 
verification 


Manufacturing 

Engineering 

•  Manufacturing 
and  inspection 
work  instructions 

•  Producibility 

•  Workflow 


Manufacturing 

Product 

•  Fabrication 

•  Assembly 
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Encourage  Quality  Focus  (Snde3of3) 

Understand/Evaluate  Key  Practices 
At  Program  Startup 

-  The  process  for  establishing  the  product  or  project  Quality 
budget 

-  Where  Quality  responsibility  is  placed  in  the  program 

-  How  quality  skills  have  been  assigned  to  the  project 

-  The  process  for  analyzing  quality  requirements  and  mitigating 
associated  risks 

-  The  quality  strategy’s  consistency  with  industry  best  practices 
Throughout  the  Life  Cycle 

-  How  management  uses  quality  data 

-  The  contractor’s  approach  for  continuous  process  improvement 

-  The  contractor’s  approach  for  preventive  and  corrective  action 

-  The  contractor's  approach  for  achieving  customer  satisfaction 

DoD  should  encourage  and  participate  with 
industry  to  apply  effective  practices  in  these  areas 


Outline 


*  How  quality  impacts  technical  risk 

*  Involving  quality  early 

*  Encouraging  a  quality  focus 

*  Incorporating  quality  in  the  systems  engineering 
technical  reviews 

*  Concluding  remarks 
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Develop  Strong  Linkages  to  the  SE 
Technical  Reviews  (TRs)  (siweiofs) 


*  The  purpose  of  these  reviews  is  to  provide  the 
program  manager  with  an  integrated  technical 
assessment  of  program  technical  risk  and 
readiness  to  proceed  to  the  next  technical  phase 
of  the  effort 

*  Process-based  technical  risks  affect  readiness  to 
transition 

-  System  engineers  develop,  manage  and  execute 
processes 

-  Quality  engineers  independently  assess  if  they  are 
working  as  designed 
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Develop  Strong  Linkages  to  the  SE  TRs  <siide2of5) 

Sharper  Focus  on  Product  Realization 
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Output 


Customer  Requirements 

•  Product  requirements 
documentation 

•  Program  cost  and  schedule 
performance  data 

•  Formal  customer 
communication 


Management  Responsibility 

•  Management  Review 

•  Quality  policy 

•  Customer  focus 

•  Risk,  issue  &  opportunity  mgmt 

•  Program  Independent  Analysis 

•  Continuous  process  improvement 
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and  Procurement 

•  Purchasing  process 

•  Purchase  order  reqts 
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Develop  Strong  Linkages  to  the  SE  TRs  (slide  3  of  s> 

Best  Practices  Check  List  Process  Description 


•  Quality  engineers/SMEs  at  the  prime  should: 

-  Tailor  quality  questions  for  each  component  of  the 
product  realization  process  to  the  specific 
situation  for  the  technical  review 

-  Pose  the  questions  to  the  quality  teams  at  the 
prime  contractor  and  the  principal  subs  during 
their  technical  reviews 

-  Assign  a  red-yellow-green  indicator  to  reflect  an 
assessment  of  the  situation  based  on  the  answer 
to  each  question 
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Develop  Strong  Linkages  to  the  SE  TRs  (sudors) 

Red-Yellow-Green  Indicator  Example 


Legend:^|  Y 

G 

U  NA 

1.  Customer 

Comments/Mitigation 

Criteria 

Has  the  local  DCMA  been  involved  and  or  briefed  on  program 
reviews,  (for  example  have  they  been  activiely  involved, 
and/or  participating  in  PMR,  SRR,  PDR's  and  FAI's  etc.)? 

Green  -  Involved  &  Integated 

Yellow  -  briefed,  little  involvsment 

Red  -  No  Involvsment 

If  GSI  is  required,  is  there  delegation  in  place? 

Does  the  program  have  a  customer  contact  plan? 

Green  -  working  to  Plan 

Yellow  -no  documented  Plan 

Red  -  No  plan 

Are  Boeing  Supplier  Quality  Source  Reps  (PQA)  identified 
and  engaged? 

Green  -  frequent  visiit  and  engaged 

Yellow  -  not  engaged  on  specific  program 

Red  -  Not  identified  and  or  engaged 

Is  the  program  measuring  customer  satisfaction?  How  Often? 

Green  -  feedback  is  avaialble  on  a  routine  basis 

Yellow  -  inconsistent 

Red  -  no  measurement 

2.  Design  for  Producibility 

Has  Quality  has  been  established  in  the  Design  review 
process 

Green  -  Yes 

Red  -  No 

Are  verifications  and  acceptance  criteria  being  incorporated 
into  the  design 

Green  -  is  yes 

Red  -  is  No 

3.  SVR/FAI 

Has  the  SVR  been  incorporated  into  the  master  schedule 
and  identified  as  a  major  milestone 

Green  -  Tier  1  thru  Tier  II  or  III  schedules  to  support 

Yellow  -  no  tier  II,  III  schedule  to  backup  major 
milstone  accomplishment 

Red  -  not 

Develop  Strong  Linkages  to  the  SE  TRs  (slide  5  of  5> 

Best  Practices  Check  List  Process  Description  cont’d 


Situation  should  be  summarized  at  prime’s  technical 
review  with  the  DoD  program  office 

-  Summary  could  be  done  using  the  eight  categories  of 
the  product  realization  process 

-  Further  aggregations  may  be  made,  e.g., 

*  Product  development 

*  Software  development 

*  Verification  and  validation 

*  Supplier  management 

*  Manufacturing 

-  Utilize  a  red-yellow-green  scale 

*  Judgment  is  needed 

*  Aggregation  should  be  based  on  organizational  needs, 
program  complexity,  etc. 

-  Mitigation  actions  underway  (if  needed)  should  also  be 
presented 


Outline 


*  How  quality  impacts  technical  risk 

*  Involving  quality  early 

*  Encouraging  a  quality  focus 

*  Incorporating  quality  in  the  systems  engineering 
technical  reviews 

*  Concluding  remarks 
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Summary 


*  Quality  engineers  /  SMEs  help  identify  and  mitigate 
process-based  technical  risks  to  a  project 

*  Industry  should  involve  quality  early  in  a  project’s  life 
cycle 

*  DoD  should  encourage  and  participate  with  industry 
on  the  application  of  effective  practices  to  encourage 
a  quality  focus  throughout  a  project’s  life  cycle 

*  Quality  engineers  /  SMEs  at  the  prime  contractor 
should  summarize  the  status  of  the  product 
realization  process  at  systems  engineering  technical 
reviews 
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Next  Steps 


•  Further  refinement  through  government  review 

-  Being  circulated  among  members  of  Quality  Advisory  Group 

•  Further  refinement  through  industry  review 

-  NDIA  Quality  Assurance  Committee  members 

-  AIA  Quality  Assurance  Committee 

-  MMA  suppliers 

•  Incorporate  into  Defense  Acquisition  Guidance 

-  How  quality  impacts  technical  risk,  involving  quality  early, 
establishing  quality  focus  material  to  be  incorporated  into  new 
Quality  Management  Section  (11.3.3) 

-  Incorporating  quality  in  the  systems  engineering  technical 
reviews  to  be  incorporated  into  section  4.5  on  SE  Execution: 
Key  SE  tools  and  techniques 
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Factors  behind  "encourage  nualitv 
focus'"  evaluation  areas 


The  Process  for  Establishing  the  Product  or 

Project  Quality  Budget 


Example  evaluation  considerations: 

*  Project  quality  administration,  product 
verification,  quality  engineering  (hardware  and 
software),  quality  planning,  and  supplier  quality 

*  Specific  quality  deliverables 

*  Capital,  equipment,  and  software  verification 
needs 

*  How  the  estimates  are  modified  when  there  are 
changes  to  the  strategy  and/or  scope  of  the 
program 

*  Measurement  technology  needs 
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Where  Quality  Responsibility  is  Placed  in  the 

Program 


Example  evaluation  considerations: 

•  Role  in  the  general  risk  identification,  classification,  and 
mitigation  process 

•  Involvement  in  the  design  change  control  and  release  process 

•  Role  in  processing  waivers,  deviations  and  engineering 
change  proposals 

•  Representation  on  Integrated  Process  Teams  and  boards  (e.g., 
change  control  board,  risk)  for  all  product  and  process 
development  activities 

•  Involvement  in  test  plans,  material  reviews,  design  reviews, 
build/buy/support  to  packages 

•  Participation  in  the  integration  of  inspection  points  into 
processing  and  test  documentation 

•  Role  in  the  supplier  management,  development, 

incentivization,  and  control  process  □ 


How  Quality  Skills  Have  Been  Assigned  to 

the  Project 


Example  evaluation  considerations: 

*  The  process  to  identify  the  need  for  quality 
management,  quality  engineering  (hardware  and 
software),  quality  planning,  supplier  quality,  and 
product  verification  skills  across  the  life  cycle 

*  The  process  to  identify  quality  skills  and  any 
associated  certifications  and  qualifications 

*  The  process  for  addressing  quality  staffing  ratios 
and  skill  shortfalls 
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The  Process  for  Analyzing  Quality  Requirements 

and  Mitigating  Associated  Risks 


Example  evaluation  considerations: 

*  The  process  for  identifying  and  achieving  quality 
tasks  in  support  of  contract  deliverables 

•  How  a  post  award  contract  analysis  for  Quality's 
tasks  was  performed  /  has  been  updated 

*  An  evaluation  of  how  the  Quality  plan  matches 
the  program  requirements  and  their  integration 
across  program  sites,  IPTs,  partners  and 
suppliers 

•  How  quality  activities  factored  into  the  Integrated 
Master  Plan  and  Integrated  Master  Schedule 
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The  Quality  Strategy’s  Consistency  with  Industry 

Best  Practices 


Example  evaluation  considerations: 

•  The  use  of  lessons  learned 

•  How  similar  programs'  quality  past  performance  have  been 
reviewed 

•  How  the  quality  plan  addresses  program  unique  processes 

•  How  plans  include  verification  approaches,  nonconformance 
handling,  operator  verification  manufacturing  self- 
examination,  nondestructive  inspection,  manufacturing 
systems,  measurement  approach,  special  measuring  and  test 
equipment 

•  Adequacy  of  the  quality  plan  to  address  all  other  program 
plans  (manufacturing,  systems  engineering,  subcontract 
management,  delivery, ...) 

•  Periodic  review  and  update 

•  Early  involvement  in  the  program 
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How  Management  Uses  Quality  Data 


Example  evaluation  considerations: 

*  Audit  needs  and  addressing  audit  findings 

*  The  process  for  analyzing  and  performing  trend 
analysis  of  internal/external  audit  findings 

*  How  quality  is  defined,  measured,  analyzed,  controlled, 
and  used  to  drive  management  decisions  and  actions 
on  the  program 

-  The  process  for  developing  and  identifying  requirements 
for  quality  metrics  and  measurement  systems 

-  The  system  for  monitoring  supplier  performance, 
including  their  product  development  activities 

-  The  process  for  review  and  update 
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The  Contractor's  Approach  for  Continuous 

Process  Improvement 


Example  evaluation  considerations: 

*  Baldridge  business  model 

*  CMMI 

*  Lean 

*  Six  sigma 

*  ISO  recertification 

*  Actions  taken  to  address  feedback  from 
assessments  performed 
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The  Contractor’s  Approach  for  Preventive 

and  Corrective  Action 


Example  evaluation  considerations: 

*  The  process  for  addressing  test  and  inspection 
findings  and  discrepancies 

*  The  process  for  addressing  supplier  nonconformances 

*  Establishment  and  maintenance  of  a  closed  loop 
corrective  action  system  that  includes  the  reporting, 
root  cause  analysis,  and  implementation  of  actions 
necessary  to  correct  and  preclude  recurrence  of 
problems,  failures,  quality  issues, 
defects/nonconformances 

*  The  process  for  using  lessons  learned  to  drive 
continuous  improvement 
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The  Contractor's  Approach  for  Achieving 

Customer  Satisfaction 


Example  evaluation  considerations: 

*  The  process  to  collect,  monitor,  and  analyze 
information  for  measuring  customer  satisfaction 

*  The  process  to  rapidly  mitigate  customer 
concerns 

*  The  process  to  communicate  with  customers  at 
all  levels 

*  The  process  /  organizational  structure  for 
reacting  to  customer  inquiries  and  needs 
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Extract  from  SE  TR  Questions 
Systems  Engineering 


Determine/evaluate  the  process  for  identifying,  allocating, 
and  documenting  product  requirements  including  safety 
and  environmental  requirements 

Determine/evaluate  the  configuration  management 
approach  for  hardware 

Describe/evaluate  how  verification  plans  are  being 
developed  for  new  products  and 
manufacturing/measurement  processes 

Describe/evaluate  how  trade  studies  been  completed,  and 
are  incorporated  into  the  design  baseline 

Identify  and  assess  the  overall  Risk  Management  Program 
and  determine  if  it  is  adequately  documented 


Extract  from  SE  TR  Questions 
Software  Engineering 


Determine/evaluate  the  processes  and  equipment  for  performing 
software  unit  testing,  integration  testing,  software  qualification, 
regression  testing,  and  operational  testing 

Determine/evaluate  the  software  acceptance  process  for  assuring 
deliverable  software  complies  with  contractual  requirements,  and 
that  all  discrepancies  and  nonconformances  are  properly 
documented  and  resolved 

Determine/evaluate  the  processes,  procedures,  and  tools  for 
performing  software  quality  assurance,  software  quality 
assessments,  verification,  validation,  configuration  control, 
metrics,  management  review,  audits,  etc. 

Determine/evaluate  the  process  for  determining  the  quality 
requirements  for  non-deliverable  software 

Determine/evaluate  the  root  cause  analysis  and  corrective  action 
process  for  software  quality  issues  n=| 


Extract  from  SE  TR  Questions 
Integrated  Product  Definition 


•  Describe/evaluate  the  verification  plans  for  "non-tooling" 
concepts  (e.g.,  determinant  assembly,  self  locating  parts, 
etc.);  describe/evaluate  how  verification  is  accomplished 

•  Describe/evaluate  the  process  for  integrating  the 
verification  requirements  into  the  work  instructions 

•  Describe/evaluate  the  process  that  identifies  key 
characteristics  (i.e.,  the  features  of  a  material,  process,  or 
part  whose  variation  has  a  significant  influence  on  product 
fit,  performance,  service  life,  or  producibility) 

•  Determine/evaluate  the  process  for  integrating  verification 
requirements  and  key  design  characteristics 

•  Describe/evaluate  the  process  for  design  evolution 
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Extract  from  SE  TR  Questions 
Product  Support 


*  Describe/evaluate  any  field  support  requirements 
for  quality  (hardware/software) 

*  Determine/evaluate  if  there  are  any  post¬ 
production  considerations  or  topics,  such  as 
spares,  technical  bulletins,  support  equipment, 
modifications/retrofit  that  must  be  considered 

*  Describe/evaluate  any  training  requirements  for 
quality  such  as  simulators 
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Extract  from  SE  TR  Questions 
Test  Engineering 


•  Determine/evaluate  the  plan  and  process  for 
conducting  verification  and  validation  including 
those  accomplished  for  Spares,  Kits  and  Support 
Equipment 

•  Describe/evaluate  the  processes  to  validate  that 
tests  adequately  assess  the  requirements  such 
as  interface  definition  and  control 

•  Describe/evaluate  the  contractual  test  plan 
requirements,  if  applicable 

•  Describe/evaluate  how  verification  and  test  plans 
were  developed  concurrently  with  the  design 
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Extract  from  SE  TR  Questions 
Supplier  Management  and  Procurement 


Describe/evaluate  the  process  for  defining  and  flowing  down  to 
suppliers  the  quality  requirements,  special  processes,  tools, 
design  criteria  and  specifications 

Describe/evaluate  how  quality  requirements  are  incorporated  into 
supplier  Statements  of  Work  (SOW)  and  Supplier  Data 
Requirements  Lists 

Determine/evaluate  the  process  being  used  to  transfer  work  to 
other  sites  and  to  define  the  quality  requirements  for  this  work 

Describe/evaluate  any  partnering  arrangements  with  other 
suppliers/customers  that  impact  quality  (hardware/software) 

Describe/evaluate  how  the  program  utilizes  the  company's 
preferred  supplier  certification  process 


Extract  from  SE  TR  Questions 
Manufacturing  Engineering 


*  Describe/evaluate  the  configuration  management  (including 
change  control)  process 

*  Describe/evaluate  the  current  documented  Geometric 
Dimensioning  and  Tolerancing  (GD&T)  approach  (Common 
datums  being  utilized,  and  datum  management) 

*  Describe/evaluate  the  program  approach  for  verifying  that 
prescribed  manufacturing  methods  have  produced  an  item 
conforming  to  engineering  drawings,  planning,  purchase 
order,  engineering  specifications  and/or  other  applicable 
design  documents 

*  Describe/evaluate  the  process  for  validation  and  periodic 
verification  of  production  tooling 

*  Describe/evaluate  the  process  for  determining  the 
capability  of  processes  and  measurement  systems  (e.g., 
Process  Capability,  Measurement  Systems  Analysis,  SPC); 
Describe/evaluate  the  mitigation  plan,  if  applicable 
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Extract  from  SE  TR  Questions 
Manufacturing  Product 


*  Determine/evaluate  the  process  for  identifying, 
documenting  and  segregating  nonconforming  materials 

*  Determine/evaluate  the  process  for  identifying  and 
controlling  purchased  items  awaiting  test  and  inspection, 
those  which  have  been  accepted,  those  which  have  been 
rejected,  and  those  awaiting  material  review  action 

*  Describe/evaluate  tests  and  inspections  used  as  a  basis  for 
government  acceptance  of  contract  end  items  and  the 
Acceptance  Test  Procedure  process 

*  Describe/evaluate  the  program  for  the  control  of  limited-life 
items  in  production  and  storage  areas 

*  Determine/evaluate  the  adequacy  and  implementation  of 
test  and  inspection  procedures 
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The  Human  As  a  Key  Consideration  in  SoS 


Several  items  of  note  in  the  US  AF  Science  Advisory  Board  (SAB) 
Report 1  referring  to  the  human  in  the  equation: 

-  “An  effective  system-of-sy stems  will  promote  collaborative  decision¬ 
making  and  shared  situation  awareness  amongst  the  human 
operators.” 

-  “...we  should  recognize  that  human-to-human  and  human-to-system 
interactions  would  continue  to  be  critical  components  of  effective 
svstems-of-svstems. 

Research  in  human  system  interaction  and  decision-making  is  required 
to  understand  better  how  to  integrate  these  elements  into  an  effective 
system-of-systems  architecture. 

-  Systems  Engineering:  human  machine  interface  (HMI)  was  the  focus 
for  a  single  system;  human-to-system  and  human-to-human 
interaction  is  the  focus  for  SoS 


*l/S  AFSAB,  Executive  Summary  and  Annotated  Brief,  SAB-TR-05-04,  July  2005 
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“We  have  identified  four  main  factors  that  require  consideration  in  improving 
System-of-Systems  Engineering  in  the  Air  Force: 

-  The  first  of  these  were  the  need  to  include  human  system  interaction  as  a 
part  of  Svstem-of-Svstems  Engineering 

-  . The  last  is  the  need  to  incorporate  discovery  learning  through 

experimentation  at  the  system-of-systems  level”1 

the  current  state  of  systems  engineering  does  not  adequately  support  the 
development  of  complex,  adaptive,  and  software-intensive  system-of-systems 
(SoS)  in  which  humans  are  parts  of  the  system.”1 

“While  significant  progress  has  been  made  with  the  network  dimension,  only 
preliminary  work  to  scope  the  requirements  for  the  human  dimension  of  NCW 
has  been  undertaken.  The  cumulative  effort  required  to  realise  the  human 
dimension  of  NCW  could  well  outstrip  the  more  readily  understood  network 
aspects  of  NCW.”  “...[A  key  goal  is  to]  Explore  the  human  dimensions  of  the 
networked  force  and  initiate  changes  in  doctrine,  education  and  training  with 
appropriate  support  mechanisms.”2 


1US  AFSAB,  Executive  Summary  and  Annotated  Brief,  SAB-TR-05-04,  July  2005 

2  NCW  Roadmap,  Australian  Government,  Department  of  Defence,  2005 
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Net  Centric  Operations  Imply 
H  Leverage  SoS  in  new  and  different  ways 

-  “Potential’  for  decision  makers  and  operators  to 

•  Unprecedented  access  tolnformation  and  assets  over  the 
network 

•  More  effective  and  efficient  human  “networking” 

•  Faster  and  more  effective  resource  utilization 

•  Faster  versus  smarter  decisions? 

Engineering  for  the  Human  -In-The-Loop  (HITL)  in  a  SoS 

-  Evolving  area  of  research 

•  But,  implementation  ahead  of  application  of  theory 

-  e.g.,  in  NCO  -  more  sources,  more  deconfliction,  more  sensemaking 
required 

•  Solution:  Historically,  engineer  through  experimentation 

•  Dilemma:  But  what  do  we  need  to  measure  and  for  what 
purpose? 
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Implication  from  DOTMPLF  on  System  E 


•  NetCentric  SoS  implies  breaking  organizational  barriers 

•  The  ley  human  element  here  is  Trust 

-We  must  not  only  re-engineer  our  system  to  leverage 
NetCenricity,  we  must  also  re-engineer  the  enterprise 

•  Will  the  system  from  organization  A  be  there  to  support  the 
system  from  organization  B? 

•  Today  -  Mismatch  between  rate  of  applying  technology 
to  the  problem  versus  the  organizational  and  business 
implication  of  the  transformation 

-All  facets  pf  DOTMPLF  must  be  re-evaluated  when  a  new 
capability  must  be  assessed  against  NetCentric  principles 
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Observations  on  Human  Systems  Integrate 


•  “In  HSI,  decision  makers  and  facilitators  take  advantage  of  technological 
developments  in  systems  engineering  and  systems  management.  Inherent 
in  several  of  these  advances  is  the  capability  to  quantify  and  measure 
human  characteristics.  These  newer  methods  also  allow  better  decisions 
to  be  made  early  in  the  design  and  development  process  where  changes 
are  relatively  inexpensive  to  make.”1 

-  Supports  the  broad  implications  of  DOTMPLF  in  Capability  Development 

-  Need:  Greater  focus  on  HITL  as  a  measurable  component  of  capability 
MOE  and  its  implementation  in  SoS  (systems  +  humans)  MOP 

H Trend:  Systems  and  products  that  can  be  operated  and  repaired  by  fewer 
people,  by  lesser  skilled  people,  and/or  people  with  lesser  training  will  be 
in  greater  demand. 

•  Manpower,  personnel,  and  training  (example:  UAV  and  takeoff/landing 
expertise)  are  becoming  key  consideration  in  cost  effectiveness  and 
mission  effectiveness 

•  Need  to  make  the  human  component  an  "inherent  part  of  the  system,"  and 
the  drive  toward  "quantification  of  people  variables"  in  the  overall  system 
engineering  of  the  system  or  SoS 


1  “Handbook  of  Human  Systems  Integration”,  Harold  Booher,  Wiley  &  Sons,  2005 
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Observations  on  Human  Systems 


NetCentric  SoS  -  The  human  is  now  taking  an  active  and  leading  role  in 
combining  systems  to  provide  new  capabilities  (at  run  time  versus  at 
design  tine) 

-  How  that  is  done  and  how  effectively  is  now  an  art. 

-  Can  we  bring  science  to  this? 

B  If  we  are  passing  Power  to  the  Edge,  we  have  a  new  training  paradigm  with 
new  SoS  assets  to  configure  and  use. 

B  Commanders  are  challenged  to  plan  tasks  in  hours  vs.  days;  planning  in 
minutes  versus  hours 

•  Paradigm  shift:  Less  decision  and  information  flow  up  and  down  the 
chain  of  command 

•  Commander  now  “shepherds”  or  “monitors”  versus  “commands” 

•  What  is  the  minimum  information  required  to  make  decisions  at  the 
Edge? 

-  How  do  we  capture  and  analyze  the  impact  of  an  operational  architecture, 
and  its  complementary  system  architectures,  when  we  are  asked  to 
accommodate  responsive,  agile,  dynamic  (on-the-fly  changes  to) 
operational  approaches  in  a  NetCentric  environment? 

“....it  can  be  expected  that  HSI  activities  will  become  more  closely 
associated  with  constructive,  virtual,  and  live  simulations”  1 

-  Measurement  of  Human-in-the-loop  parameters  (primarily  cognitive  and 
social  parameters)  has  been  problematic  for  SEs  to  define  and  measure 

-  Experimentation,  in  lieu  of  engineering,  has  been  pursued 


1  “ Handbook  of  Human  Systems  Integration”,  Harold  Booher,  Wiley  &  Sons,  2005 
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“...the  character  and  performance  of  a  C2  system  may  change  as 
anyone  of  elemehts  in  these  three  categories  changes.” 1 

“J. since  the  iuman,  organizational  and  technological  elements  are 
closely  linked  in  most  cases,  optimizing  each  one  of  them  at  a  time 

under  ceteris  paribus  [other  things  being  equal]  assumptions  for  the  other 
two  rarely  ever  results  in  an  efficient  C2  system.”2 


Humans 


Organization 


Technology 


1  NATO  Code  of  Best  Practice  for  C2  Assessment,  CCRP,  2002 

2  Schot,  J.,  &  Rip,  A.  (1996).  The  past  and  future  of  constructive  technology  assessment. 

In  Technological  forecasting  and  social  change,  54,  251-268.  New  York:  Elsevier  Science.  [Chapter  6] 
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Observation  from  the  Perspective  of  Operations  Analyst 


Pearls  of  Wisdom1 

-  We  can  no  longer  expect  to  "bend"  people  to  technology;  rather,  we 
need  to  study  how  best  to  produce  creativity  at  the  nexus  of  people 
and  technology 

-  Needs  are  not  always  task  related 

-  Needs  may  be  cognitive,  behavioral,  or  social,  such  as  how 
information  is  displayed,  how  teams  operate,  how  tasks  are  shared 

-  We  must  recognize  the  importance  of  relationships  [read  “trust”] 

-  Technical  solutions  cannot  replace  human  judgment 


4 

MORS  Workshop  Report:  How  Cognitive  and  Behavioral  Factors  Influence  Command  and  Control, 
Military  Operations  Research  Society,  22  April  2005 


NDIA  -  Dr.  Abe  Meilich 


Copyright  2006  Lockheed  Martin  Corporation,  All  Rights  Reserved 


10 

10/24/06 


Observation  from  the  Perspective  of  Operations  Analyst 


•  Pearls  of  Wisdom1  (cont’d) 

-  Decision  making  tasks 

•  How  much  information  is  enough  to  make  a  decision? 

•  The  lower  the  tolerance  for  risk,  the  higher  the  demand  for 
information  to  avoid  that  risk 

•  Commanders  manage  information  differently,  therefore, 
information  must  be  shaped  for  the  individual  commander 

-  Tasks  do  not  always  require  quantifiable  information 

-  Just  because  something  cannot  be  measured  or  quantified,  doesn't 
mean  it  isn't  important 

•  Qualitative  methods,  such  as  observation,  have  their  uses  as  well 

-  Commanders  must  perform  their  tasks  in  a  timely  manner 

•  Concern  that  they  will  wait  for  more  or  better  information  rather 
than  act  or  make  a  decision 

•  Need  to  balance  the  need  for  quick  decision  making  with  informed 
decision  making 

MORS  Workshop  Report:  How  Cognitive  and  Behavioral  Factors  Influence  Command  and  Control, 

Military  Operations  Research  Society,  22  April  2005 
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Observation  from  the  Perspective  of  Operations  Analyst 


-  On  the  Network  -  Email,  phone,  and  chat  proliferate  workload 
irrespective  of  the  chain  of  command 

-  Increased  capability  may  decrease  effectiveness  (more  technology, 
information  overload) 

-  Concern  -  In  Network-Centric  Warfare,  everything  depends  on  the 
network  -  What  if  it  doesn't  work? 


5^ 


* 

MORS  Workshop  Report:  How  Cognitive  and  Behavioral  Factors  Influence  Command  and  Control, 
Military  Operations  Research  Society,  22  April  2005,  p.22 
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System  Engineering  tradeoffs 

-  Drivers  in  the  problem  space 

•  Changing  nature  of  military  operations  (sectarian-based  urban  vs 
national  armies 

•  Call  for  certain  cognitive  and  social  behavior 

-  e.g.,  each  system  optimized  for  human  in  that  domain  -  SoS 
human  behavior  yet  to  be  characterized 

-  Requirements  in  terms  of  the  human  component  of  the  architecture 

•  As  opposed  to  mission,  task,  or  technology 

-  Examples 

•  Agility  and  adaptability  refers  to  the  human,  rather  than  the 
command  and  control  process. 

•  Distributed  collaboration  refers  to  the  people  who 
collaborate,  rather  than  the  tools  used  to  collaborate. 

-  Technology  can  work  well,  but  still  not  contribute  to  battlefield 
performance 


NDIA  -  Dr.  Abe  Meilich 
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Net  Centric  Operations  Conceptual  Framework  (NOCF) 


•  NOCF1  consist  of  the  physical,  information,  cognitive,  and  social  domains 

•  The  domains  are  inextricably  linked  in  the  NCO  environment 

•  All  the  domains  must  support  processes  carried  out  during  Effects-Based 
Operations,  if  they  are  to  be  of  any  value 

•  The  ingestion  of  information  about  the  battlespace,  conversion  into 
information,  and  creating  awareness  as  the  basis  for  action  are 
fundamental  tasks  in  the  cognitive  domain 

H  These  are  uncomfortable  attributes  for  systems  engineers  to  deal  with  in 
their  design 

0  Yet  they  are  central  to  successfully  leveraging  NCO 

•  How  do  we  design  for  the  mantra  “the  right  information  to  the  right  person 
at  the  right  place  in  the  right  form”? 

-  If  requirements  are  static,  engineering  is  straightforward 

-  Challenge:  How  do  human  decision  makers  perceive  the  actions  in  the 
physical  domain  as  reported  to  them?  And  then  how  do  they  make  the 
decision? 

-  Understanding  this  comes  down  to  how  people  process  information  and 
under  what  mental  models 

-  Complexity  and  uncertainties  are  the  norm  here 

1Net  Centric  Operations  Conceptual  Framework ,  Version  2,  OSD/OFT/NII,  June  2004  14 
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etCentric  Operational 
(NOCF) 


onceptual  Framewon 


The  human  and  organizational  components  will  influence  the 
cognitive  and  social  domains  of  the  NCOF 


Information 

Domain 


Better  Quality  Networking  and  Information  Sharing 


Ensures 


Cognitive  & 
Social  Domains 


Physical  Domain 


Improved  Information  Quality 


Improved  Shared  Awareness/Understanding 


Enhanced  Collaboration/Interactions/Decision  Making 

More  Agile  Command  and  Control 

^  Which  Contributes  to... 
More  Agile  Force  Elements/MCPs 

^  Which-  ultimately^  leads  to 

Improved 

Mission  Effectiveness  & 

Force  Agility 


MOEs 


1Net  Centric  Operations  Conceptual  Framework,  Version  2,  OSD/OFT/NII,  June  2004 
NDIA  -  Dr.  Abe  Meilich  Copyright  2006  Lockheed  Martin  Corporation,  All  Rights  Reserved 


15 

10/24/06 


Measurement  using  the  NOCF 


Information 

Sources 
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1 Net  Centric  Operations  Conceptual  Framework,  Version  2,  OSD/OFT/NII,  June  2004 
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NCOF  support  to  Future 


Past  Investments 


y 


Assessment  of  How  We  Fight  Today 
(Based  on  Evidence  &  Analysis) 


NCW  Conceptual  Framework 


"\ 


Future 

Investments 


Informs 


y 


'Net  Centric  Operations  Conceptual  Framework,  Version  2,  OSD/OFT/NII,  June  2004 
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Drivers  of  Decision-making 


1  NATO  Code  of  Best  Practice  for  C2  Assessment,  CCRP,  2002 


Least  Desirable 
Situation 


Time  Available  Proactive 


The  lower  the 
tolerance  for  risk,  the 
higher  the  demand 
for  information  to 
avoid  that  risk 


Most  Desirabl 
Situation 


NDIA  -  Dr.  Abe  Meilich 
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Challenge  of  extracting  meaningful  data  from 
experimentation  with  HITL: 

•  “Decision-makihg  that  is  rule  or  algorithmically  based  can 
be  modeled  directly,  but  error  rates  should  be  estimated  if 
humans  are  involved  in  the  relevant  decision-making 

-  Implications  for  SE:  FMEA  of  technology  based  on  effects  of 
the  HITL  on  Mission  success 

•  Operational  knowledge  of  human  issues  is  still  weak  in 
many  areas  [of  C2].  Systematic  effort  is  required  for 
organizing  a  consistent  program  for  experiments  on  human 


issues. 


”1 


1  NATO  Code  of  Best  Practice  for  C2  Assessment,  CCRP,  2002 
NDIA  -  Dr.  Abe  Meilich  Copyright  2006  Lockheed  Martin  Corporation,  All  Rights  Reserved 
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Summary  of  Approaches  for  Engineering  in  the  Cognitive 
and  Social  Domains _ _ _ 

•  Observation  and  feedback 

-  Expensive  if  done  late  in  development  during  prototyping 

-  Needs  to  be  done  very  early  to  minimize  cost  of 
development 

•  Stimulus/response  analysis 

-  Paper  simulation 
-War  gaming 

•  Concept  exploration  with  instrumentation 

•  Isolation  of  components  of  cognitive  domain  and  social 
domains 

-  Use  NCOCF  model  attributes 


NDIA  -  Dr.  Abe  Meilich 
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Summary 


•  We  need  to  provide  systems  and  services  to  our  warfighters 
quickly,  but  not  without  understanding  how  they  are  used  both 
cognitively  and  socially  on  the  battlefield 

•  This  is  broader  than  the  Human  Computer  Interface  (HCI)  used  in 
systems  engineering  today 

•  Just  as  bio-engineering  revolutionized  medicine,  Cognitive 
Engineering  and  HSI  as  key  considerations  in  systems 
engineering  are  and  will  revolutionize  the  conduct  of  on  both  the 
strategic  and  tactical  levels  of  warfare 

•  Resource  to  review  Human  Performance  Engineering 

-  ONR  »SC-21/ONR  S&T  Manning  Affordability  Initiative  (FY96- 
FYOO) 

•  http://www.manningaffordability.com/s&tweb/index.htm 


NDIA  -  Dr.  Abe  Meilich 
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Other  Observations 


NDIA  -  Dr.  Abe  Meilich 
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It  has  been  noted1  that  good  commanders  have  dealt  with  the  complexities  of 
the  battlespace  by  inserting  a  “human  in  the  loop” — whether  themselves, 
subordinate  commanders,  staff,  or  watch  personnel — to  make  complex 
decisions,  to  assess  ambiguous  information,  and  to  fill  in  the  blanks  where 
information  is  wanting. 

The  human  is  at  the  root  of  network-enabled  effects-based  operations. 

Whereas  success  in  “classic”  effects-based  approaches  largely  depended  on 
the  abilities  of  the  humans  in  the  loop  to  deal  with  the  complexity  in  their  heads, 
in  network-enabled  operations  they  need  no  longer  be  left  to  their  own  devices. 

-  Better  and  more  meaningful  support  from  networking  can  enable  decision¬ 
makers  to  bound  complexities  and  deal  with  ambiguities  better  and  thereby 
increase  the  probability  of  a  correct  decision. 


1  Complexity,  Networking  and  Effects  Based  Approaches  to  Operations,  Edward  A  Smith,  CCRP,  2006 


NDIA  -  Dr.  Abe  Meilich 
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World  War  IV  and  implications  for  SE  and  the  < 


It  was  noted  by  Maj.  Gen.  ROBERT  H.  SCALES  (ret.)  that: 

THE  EVOLUTION  OF  WARFARE 

THE  CHEMISTS'  WAR 

The  decisive  strategic  advantage  on  the  World  War  I  battlefield  was  driven  by  new  applications  of 
chemistry  and  chemical  engineering.  Germany,  for  example,  exhausted  its  supplies  of 
gunpowder  nitrates  in  1915,  but  the  synthesis  of  nitrates  by  German  scientists  allowed  the  war 
to  continue  for  another  three  years. 

THE  PHYSICISTS'  WAR 

The  atomic  bomb  ended  World  War  II,  but  exploitation  of  the  electromagnetic  spectrum  in  the 
form  of  wireless  communications  and  radar  won  it  for  the  allies. 

THE  INFORMATION  RESEARCHERS'  WAR 

In  World  War  III,  intelligence  and  the  ability  to  fully  exploit  it  allowed  the  U.S.  to 
defeat  the  Soviet  Union.  Information-age  concepts  of  transformation  and  net- 
centrism  mark  the  end  of  this  epoch. 

THE  SOCIAL  SCIENTISTS'  WAR 

To  win  World  War  IV,  the  military  must  be  culturally  knowledgeable  enough  to  thrive 
in  an  alien  environment.  Victory  will  be  defined  more  in  terms  of  capturing  the 
psycho-cultural  rather  than  the  geographical  high  ground.  Understanding  and 
empathy  will  be  important  weapons  of  war. 

1  Maj.  Gen.  ROBERT  H.  SCALES  (ret.) ,  “Clausewitz  and  World  War  IV” ,  Armed  Forces  Journal,  July  2006 
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Observations  from  Maj.  Gen.  ROBERT  H.  SCALES  (ret) 


“Machines  arid  processes  might  make  intelligence  easier  to  parse  and  read.  But 
knowing  the  enemy  better  than  he  knows  us  is  inherently  a  psycho-cultural  rather 
than  a  technological,  organizational  or  procedural  challenge.”1 


f*The  enemy  has  drawn  us  unwillingly  into  fighting  him  at  the  tactical  level  of  war 
where  the  importance  of  technology  diminishes  in  proportion  to  the  value  of 
intangibles.  “ 

“Models  of  human  cognition  can  also  be  used  to  diagnose  performance  failures 
during  simulated  exercises.  “ 

“We  are  in  a  race,  and  the  times  demand  change.  World  War  IV  can  only  be  won  by 
harnessing  the  social  and  human  sciences  as  the  essential  amplifiers  of  military 
performance,  just  as  the  physical  sciences  were  the  amplifiers  of  past  world  wars. 


“Of  course,  new  planes,  ships  and  combat  vehicles  will  have  to  be  built  to  win 
World  War  IV,  but  building  new  social,  cultural  and  learning  structures  will  have  to 
become  the  first  priority  for  resources  within  the  Defense  Department.  There  is  an 
old  saying  that  the  Navy  and  the  Air  Force  man  the  equipment  and  the  Army  and 
Marine  Corps  equip  the  man.  Surely  those  services  that  focus  on  the  man  rather 
than  the  machine  should  receive  a  disproportionate  share  of  future  defense 
budgets?  “ 

“What  will  the  new  amplifiers  be? . Will  new  human  and  behavioral 

developments  make  us  more  effective  in  battle?  Only  time  will  tell.  But  none  of 
these  questions  can  be  answered  by  speculation  alone.  The  Defense  Department 
must  invest  the  resources  now  to  realize  the  potential  of  psycho-cultural  sciences 
to  winning  World  War  IV.” 


1  Maj.  Gen.  ROBERT  H.  SCALES  (ret.) ,  “Clausewitz  and  World  War  IV” ,  Armed  Forces  Journal,  July  2006 
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Implications  of  Agent  Based-Simulation 


<  Agent  --based  simulation  (  e.g.,  SEAS)  now  takes  into 
account  the  Cognitive  Domain  (via  Programmed 
behaviors  BBehaworal  and  decision-making  logic) 

-  Good  enough  for  SE  trade  studies? 

-  Good  enough  to  go  operational? 

-  How  much  can  we  trust  the  outcomes  of  these 
simulations  where  100s  ( if  not  1000s)  of  entities  interact 
to  end  in  an  effect 


NDIA  -  Dr.  Abe  Meilich 
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Beyond  the  OOP  A1  loop 


•  OODA  is  the  foundation  for  understanding  Effects- 
Based  Operations 

•  OODA  has  different  implications  at  the  strategic  and 
tacncal  levels 

•  How  do  we  engineer  a  capability  to  improve  the  OODA 
wij hout  considering  the  HITL? 

•  Clearly,  the  cognitive  processes  that  allow  the  human 
to  perceive  and  decide  are  at  the  center  of  the  human 
dimensions  of  war 

•  Capabilities  are  written  in  terms  of  what  the  human 
needs  to  accomplish  ,  not  what  the  machine  must  do 


1OODA:  observe,  orient,  decide,  act 


NDIA  -  Dr.  Abe  Meilich 
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Step  1  for  Systems 
Engineering:  Establish  a 
\  Tangible  Requirements 
f  Management  Process 


25  Oct  06 


Jim  Miller 
727  ACSG/EN 
Phone:  (405)  736-5771 
james.c.miller@tinker.af.mil 


Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Verification 

Loop 


Requirements 

Loop 


Design 

Loop 


Design 

Synthesis 


OUTPUTS 


Purpose 


•  Published  Operating  Instruction  to  provide: 

-  An  organization-wide  process  to  ensure  the 
development  of  concise,  quantifiable,  and 
unambiguous  requirements 

-  Guidance  for  a  requirements  management  process 
including  the  preparation,  update,  and 
maintenance  of  requirements  documentation  to 
support  the  program  office 

-  Means  to  implement  a  consistent  application  of  a 
disciplined  systems  engineering  process  for  the 
management  of  requirements 


Purpose  (Cont) 


•  Is  an  iterative  process 

•  Intended  for  the  working  level 

•  Orderly,  team-oriented  process  involving: 

-  Project  engineers  -  Contractors 

-  Program  managers  -  Users/Customers 

•  Requirement  process  to  work  with 
contractor’s  processes/timelines — not 
duplicative 

•  Documented  output  via  Requirements 
Correlation  Matrix  (RCM) 


Requirements  Mngt  Process  Flowchart 


Step  1 :  Receive  Req’s 


*  Receive  Approved  and  Funded 
Requirements  Document: 

-  Formally  starts  this  process 

-  Document  can  be  anything  that  is  recognized 
as  “official”  by  the  Program  Manager 

•  Contract 

•  SOW/PWS 

•  AF  Form  1067 

•  Email 

-  OPR:  Program  Manager 
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Step  2:  Build  IRT 


*  Build  Integrated  Requirements  Team  (IRT): 

-  Formally  established  in  writing 

-  Will  include  at  a  minimum: 

•  Project  Engineer 

•  Program  Manager 

•  Representative(s)  from  the  User 

•  Representative(s)  from  the  contractor 

-  Each  member  there  to  ensure  entire  IRT: 

•  Fully  understands  each  requirement 

•  Concurs  with  all  derived  requirements 

•  Quantifies  each  requirement 

•  Provides  detailed  operational  uses/cases  for  each  req 

-  Intended  to  meet  regularly 


-  OPR:  Program  Manager 


7 


Step  3:  ID  New  Req’s 


*  Identify  and  Extract  New  Requirements: 

-  Identify,  extract  and  list  all  requirements  from 
the  requirements  document  in  Step  1 

-  Identify  the  specific  reference  (e.g.  paragraph 
and  document) 

-  OPR:  Project  Engineer 
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Step  4:  RCM 


•  Fill  Out  Requirements  Correlation  Matrix  (RCM): 

-  Document  in  a  software  application  to  allow  for  future 
updates  (e.g.  spreadsheet) 

-  Intended  Fields: 

•  Requirement  title 

•  Requirement  source  (specific  reference,  e.g.  para  &  doc) 

•  Derived  Requirements  (logical  breakdown  of  req) 

•  Brief  requirement  definition  (1-3  sentences  only) 

•  Quantification  of  the  requirement  (this  is  key) 

•  Operational  Assessment  (all  scenarios  and  uses) 

•  Initial  Risk  Assessment  (e.g.  red/yellow/green) 

-  Review  any  lessons  learned  from  previous  teams 


-  OPR:  Project  Engineer 
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Step  4:  RCM  (Cont) 


Each  IRT  member  is  a  SME 


-  Requirement  title 

-  Requirement  source 

-  Derived  Requirements 

-  Brief  requirement  definition 

-  Quantification  of  the  requirement 

-  Operational  assessment 

-  Initial  risk  assessment 


Step  5:  Build  Metrics 


•  Build  Metrics: 

-  Two  Metrics: 

*  Requirement  Management  Metric  (i.e.  a  snapshot) 

*  Requirement  Growth  (i.e.  a  trend) 

-  Must  be  updated  throughout  process 

-  Must  be  shown  regularly  to  management 

*  Quarterly  Weapon  System  Review 

*  Once  a  Month  Staff  Meeting 

*  Any  significant  event  (e.g.  PMR,  PDR,  CDR,  TIM,  ...) 


-  OPR:  Project  Engineer 
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Requirement  Management  Metric 

Example  2 


Requirements 

Reviewed 

Requirements 

Quantified 

Requirements 

Changed 


As  of:  Date  2 


Total  Requirements  =  Requirements  +  Derived  Requirements 


Requirement  Growth 

Example  2 


Date  2 


Date  3 


Date  4 


Step  6:  ID  Use  Cases 


*  Identify  Operational  Scenarios: 

-  Clearly  list  all  operational  uses  and  scenarios 

-  Done  for  each  req  and  derived  req 

-  Necessary  to  ensure  design: 

•  Encompasses  all  user’s  intentions 

•  Helps  identify  all  potential  special  cases 

•  Comprehensive  list  of  what  to  test/measure 


-  OPR:  Integrated  Requirements  Team 
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Step  7:  Derived  Req’s 


*  Identify  and  Extract  Derived  Requirements: 

-  This  is  an  iterative,  spiral  process 

-  Expect  derived  requirements  to  increase/clarify 
as  user  discusses  all  operational  uses 

-  Key  is  not  to  “assume”  anything 


-  OPR:  Integrated  Requirements  Team 
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Step  8:  Quantify 


•  Define/Clarify/Quantify  Requirements: 

-  The  heart  of  the  RCM 

-  Specific,  unambiguous  quantification  of  each  req 

*  Key  for  later  testing 

*  Provide  agreed  on  pass/fail  criteria 

*  Not  open  to  interpretation 

-  Resolve  all  “TBDs” 

-  Document  in  minutes  how  quantification 
determined  on  requirements  and  indication  of 
concurrence  from  all  members 

-  OPR:  Integrated  Requirements  Team 


Step  9:  Update  RCM 


•  Update  RCM: 

-  Update  with  each  spiral  from  the  IRT 

-  Serves  as  the  “common  sheet  of  music”  for 
the  IRT 

-  This  is  the  source  of  data  for  the  metrics 


-  OPR:  Project  Engineer 
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Step  10:  Req  Review 


Unbiased  Review: 

-  Purpose  to  validate  the  RCM 

•  Sufficient  level  of  detail  via  derived  requirements 

•  Proper  quantification  of  each  requirement  __ _ 

•  Strong  review  of  non-quantified  and  TBD  req’s  ^  I 

•  Comprehensive  review  of  use  cases  and  what-ifs^  . 

-  Project  Engineer  presents  the  RCM  * 

to  review  board  ■  ^ 

-  Chief  Engineer  is  the  Chair 

-  Other  members  are  SMEs  and  experienced  personnel 

-  Members  from  both  inside  and  outside  organization 


-  OPR:  Chief  Engineer 
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Step  11:  Changes 


Requirements  Changes: 

-  Project  Engineer  takes  Review  Board  results 
back  to  IRT 

•  Requests  for  clarification 

•  Recommendations 

•  Corrections 

-  Can  also  result  from  IRT  meetings  or  program 
events  (PDR,  CDR,  PMRs,  etc..) 


-  OPR:  Project  Engineer 
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Step  12:  RCM  to  Test 


•  Pass  RCM  to  Test  Team: 

-  Test  Team  uses  this  a  core  input  to  build 
successful  test  program: 

•  Are  the  quantified  requirements  testable 

•  Determine  method  of  testing 

•  Awareness  of  problem,  or  risk,  areas 

•  What  resources  are  needed 

•  Ensure  test  plan  is  comprehensive 

•  How  handle  non-quantified  requirements 

-  Another  Ol  is  in  work  to  describe  this  process 


-  OPR:  Project  Engineer 
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Step  13:  RCM  Risk 


•  Pass  RCM  to  Risk  Team: 

-  Risk  Management  Team  uses  this  a  key  input 
for  risk  management  plan: 

•  Review  all  requirements  the  IRT  initially  flagged  as 
red  or  yellow 

•  What  resources  are  needed  to  mitigate 

•  Ensure  risk  plan  is  comprehensive 

•  Incorporate  into  program  schedule 

-  Another  01  is  in  work  to  describe  this  process 


-  OPR:  Project  Engineer 
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Step  14:  Maintain  RCM 


Maintain  and  Track  RCM: 

-  Have  regular  IRT  Meetings  to  continually 
update  RCM 

•  Eliminate  TBDs 

•  Review  quantification 

•  Ensure  have  all  operational  uses/scenarios 

•  Ensure  included  all  derived  requirements 


-  OPR:  Program  Manager 
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Step  15:  Report 


*  Report  Metrics: 

-  Regularly  update  metrics 

-  Ideal  Goal  is  100%  reviewed  and  100% 
quantified 


-  OPR:  Project  Engineer 
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Step  16:  Lessons  Learned 


Document  Lessons  Learned: 

-  Develop  a  place  for  the  organization  to  log  this 
(if  not  have  already) 

-  Good  place  to  start  developing  “standard” 
quantification  to  common  requirements 

•  Light  brightness  on  aircraft 

•  Noise  levels  on  headphones 

•  VTC  standards 


-  OPR:  Project  Engineer 
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Typical  Schedule 


•  Show  the  Flow  days  here 

-  Recommend  basing  on  Requirement  receipt 
plus  days 
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Basic  Systems  Engineering  Process 


Requirements 

Analysis 


Verification 

Loop 


Requirements 

Loop 


Functional 

Analysis/ 

Allocation 


Analysis  & 
Control 


Design 

Loop 


Design 

Synthesis 


OUTPUTS 
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Detailed  Systems  Engineering  Engine 


Stakeholder  Inputs 
Mission  scenarios 
CONOPS 
MOEs 

Environments 
Constraints 
COTS/GOTS/BOTS 
Re-Use  and 
Legacy 

Interoperability 
Supportability 
Technology  base 
Program  decisions 

Previous  phase 
results 

Requirements  applied 
through  standards 


Requirements  analysis 

Analyze  missions  and  environments 
Identify  functional  req’s 
Define/refine  performance  req’s 
Define/refine  design  constraint  req’s 
Validate  req’s  with  stakeholders 


-► 


.  Recj  u  i  re  me  n  ts  _  Loop  _ 


J 


System  Analysis 

Modeling  &  Simulation 
Trade  studies  (Affordability,  etc.) 
Effectiveness  analyses 
MOEs 


f. 


Functional  analysis  /  allocation 

Decompose  to  next  lower-level  functions 
Allocate  performance  and  other  limiting 
requirements  to  next  functional  level 
Define/refine  functional  interfaces 
.  (internal/external)  , 

\  Define/  refine/  integrate  functional  architecture  J 

•• ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 

Design  Loop 


Verification  Loop 


Synthesis 

Transform  each  level’s  architecture  (functional  to 

physical) 

Identify  candidate  architecture  styles 
Define  alternative  system  concepts  and  configuration 

items 

Define/refine  physical  interfaces  (internal/  external) 
Select  “best  value”  design  architecture  based  on  trade 


Document 

Lessons 

Learned 


Project  Engineer 
Program  Manager 
Chief  Engineer 
IRT 


C^AnaK/7P  mission^ 


St?' voider  Inputs 

Mission  scenarios 

CONOPS 

MOEs 

Environments 
Constraints 
OCTC/CCTG/oOTS 
Re-Use  and  Legacy 
Interoperability 
Supportability 
Technology  base 
Program  decisions 
Previous  phase  results 
Requirements  applied  through 
standards 


uenneppgfine  performance 
Defi  ■  ■ 


M\ 


System  Analysis 

Modeling  &  Simulation 
Trade  studies  (Affordability,  etc. 
Effectiveness  analyses 
MOEs 


Functional  analysis  /  allocation 

Decompose  to  next  lower-level  functions 
Allocate  performance  and  other  limiting  requirements  to  next 
functional  level 

Define/refine  functional  interfaces  (internal/external) 
Define/  refine/  integrate  functional  architecture 

Design  Loop 


Verification  Loop 


Synthesis 

Transform  each  level’s  architecture  (functional  to  physical) 
Identify  candidate  architecture  styles 
Define  alternative  system  concepts  and  configuration  items 
Define/refine  physical  interfaces  (internal/  external) 
Select  “best  value”  design  architecture  based  on  trade  studies 


Process  Outputs 


Wi 


Questions? 
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Detailed  Systems  Engineering  Functional  Instructions 


Systems  Engineering  Process 


Expand  Collapse  Previous  Next 

Launch  No.  Title 


Create 


000  General  -  Systems  Engineering 

5  WH-SE-000.01  PRO-3751  Systems  Engineering  Process  Documents 

[5  WH-SE-0QQ.02  PRQ^6188  Define  and  Manage  Product/Service  Requirements 
5  WH-SE-QQQ  03  PRO-6189  Plan  and  Control  Product/Service  Development  and  Definition 
5  WH-SE-Q00.Q4  PRO-6190  Concurrently  Develop  and  Define  Products/Services 
5  WH-SE-QQQ .05  PRO-6191  Verify  &  Validate  Products/Services 

5  WH-SE-QQ0.Q6  D950-10459-1  Common  Systems  Engineering  Process  Framework  (CSEPF) 
5  WH-SE-000.07  D950-1 0446-1  Systems  Engineering  Process  Manual  (SEPM) 

MOO  Manage  Product  Traceability 
^  WH-SE-100  Manage  Product  Traceability 

5  WH-SE- 100.01  BPI-3472  Manage  Product  Requirements  Traceability 

^  WH-SE- 101  Requirements  Traceability 

^  WH-SE-FI .102  Product  Requirements  Traceability  at  IDS  Wichita 

W20O  Define  and  Analyze  System  Requirements 

^  WH-SE-200  Define  and  Analyze  System  Requirements 

|p  WH-SE-200. 01  BFI-3471  Manage  Requirements 

|5  WH-SE-200 .02  BFI-3473  Perforin  Systems  Analysis 

[5  WH-SE-200 .03  BFI-3433  Define  and  Develop  Requirements 

^  WH-SE-201  Requirements  Analysis 

^  WH-SE-202  Requirements  Parse 

^  WH-SE-203  Requirements  Allocation  Sheet 

WH-SE- 2 04  Prepare  Specifications 
WH-SE-206  System  of  Systems  Engineering  Process 
WH-SE-207  System/Seqment  Specification  Template 
yL,  WH-SE-209  System/Subsystem  Specification  DID  (DI-IFSC--81431A) 

^  WH-SE-210  Analyze  and  Define  Hardware  Requirements 

^  WH-SE-211  Hardware  Requirements  Documentation 

WH-SE-212  Nuclear  Hardness  and  Survivability  -  Requirements  Flowdown 
v300  Define  and  Analyze  System  Functions 
^  WH-SE-300  Define  and  Analyze  System  Functions 

5  WH-SE-300  01  BPI-3479  Define  and  Develop  Functions 

^  WH-SE-301  Define  System  States  and  Modes 

W  WH-SE-302  Functional  Analysis  Allocation 


^400  System  Design  Synthesis 

^  WH-SE-400  Perform  System  Design  Synthesis 

|5  WH-SE-400. 01  BFI-3464  Develop  Integrated  Design 

|5  WH-SE-400. 02  BFI-3468  Integrate  System 

WH-SE-402  System  Design  Procedure 

^  WH-SE-403  Manage  Technology  and  Product  Line  Evolution 

^  WH-SE-404  Technology  Management 

^  WH-SE-405  Tailoring  the  Systems  Engineering  Process 

^  WH-SE-407  Operational  Concept/Mission  Analysis/Measures  of  Effectiveness 

^  WH-SE-408  Define  System  States  and  Modes 

^  WH-SE-410  Technical  Performance  Measurement 

Q  WH-SE-411  Technical  Performance  Measures  Template 

^500  Plan  System  Verification  and  Validation 

^  WH-SE-500  Plan  System  Verification  and  Validation 

v600  Manage  Interfaces 

^  WH-SE-600  Manage  Interfaces 

5  WH-SE-600. 01  BFI-3470  Manage  Interfaces 

^  WH-SE-601  Interface  Design/Control  Procedure 

WH-SE-603  ICP  No.  208,  Strategic  Systems  Interface  Control  Plan 

T700  Perform  Verify  and  Validate  Systems 

|  WH-SE-700  Validate  System 

|  WH-SE-70Q.01  BPI-3475  Perform  Verification 

f  WH-SE-700.02  BPI-3474  Perform  Validation 

T900  Miscellaneous 

^  WH-SE-901  Simulation  Performance  Analysis  Procedure 

^  WH-SE-904  Human-System  Integration  Process 

H  WH-SE-905  System  Safety  Program  Implementation,  Management  and  Control 
^  WH-SE-906  System  Safety  Process 

^  WH-SE-907  System  Safety  Analysis 

W  WH-SE-908  JointTasks  ■  Safety,  Health  &  Environmental  Affairs  (SHEA)  Support  Procedure 


^100  Manage  Product  Traceahilitv/ 

W:  roc-  iuu  Manage  Product  Traceability 
p  WH-SE- 100.01  BPI-3472  Manage  Product  Requirements  Traceability 

W  WH-SE- 101  Requirements  Traceability 

^  Wn-oE  r:.4  00  Product  Requirements  Traceability  at  ins  wi^»- 

^400  System  Design  Synthesis 

^  WH-SE-400  Perform  System  Design  Synthesis 

fp  WH-SE-400.01  BPI-3464  Develop  Integrated  Design 

|P  WH-SE-400. 02  BPI-346S  Integrate  System 

WH-SE-402  System  Design  Procedure 
WH-SE-403  Manage  Technology  and  Product  Line  Evolution 
^  WH-SE-404  Technology  Management 

^  WH-SE-405  Tailorina  the  Svstems  Enaineerina  Process 

M  WH-SE-407  Operational  Concept/Mission  Analysis/Measures  of  Effectiveness  — S 


a 


-202 

WH-SE- 2  04 
WH-SE-206 
WH-SE-207 
WH-SE-209 
WH-SE-21 0 
WH-SE-21 1 
WH-SE-21 2 


lUUlimiiUils  AhaIVSIS 

Requirements  Parse 
Requirements  £f!ocaflon  sneer 

Prepare  specmcanons 

System  of  Svstems  Engineering  Process 

System/Segment  Specification  Template 


System/Subsystem  Specification  DID  (DI-IPSC--81431  A) 

Analyze  and  Define  Hardware  Requirements 

Hardware  Requirements  Documentation 

Nuclear  Hardness  and  Survivability  -  Requirements  Flowdown 
T300  Define  and  Analyze  System  Functions 

^  WH-SE-300  Define  and  Analyze  System  Functions 

|P  WH-SE-300  01  BPI-3479  Define  and  Develop  Functions 
^  WH-SE-301  Define  System  States  and  Modes 

W  WH-SE-302  Functional  Analysis  Allocation 


Implementing  Systems 
Engineering  in  a 
Sustainment 
Environment 
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So  What  is  the  Problem? 


High-level  policy  is  a  good  and  necessary  first 
step,  however,  more  detailed  direction  is 
essential  to  turn  the  policy  into  a  workable, 
grass-roots  program 

Sustainment  different  than  acquisition 


Our  organization  had  insufficient  direction, 
documentation,  and  procedures  to  begin 


implementing  an  effective,  comprehensive 
Systems  Engineering  program 


iO 


So  What  Are  Doing  About  It? 


•  Instigated  a  step-by-step  approach  to  begin 
implementing  systems  engineering  throughout 
the  sustainment  organization 

•  Tangible  approach  that  is: 

-  Aimed  at  the  working  level 

-  Affects  all  phases  of  a  program’s  lifecycle 

-  Applicable  throughout  entire  organization 

-  Accounts  for  organization’s  progress  through  metrics 


Phase  1 :  Awareness  of  Need 


Phase  2 
Phase  3 
Phase  4: 
Phase  5 
Phase  6 
Phase  7 


Workforce  Training 
Identify  Applicable  Programs/Orgs 
Identify  and  Define  Processes 
Incentivize  Contractors/Partners 
Develop  Library  of  Tools 
Track  Progress  via  Metrics 


Phase  1 :  Awareness  of  Need 


Brief  senior  leadership  and  got  documented  buy-in 
S  Identify  focal  point  for  organization 
S  Prepare  SE  re-invigoration  briefing 
S  Present  briefing  to  all: 

Supervisors  (Squadron  Commanders/Directors) 

S  Program  Manager’s 
^  Logisticians 
Engineers 

Equipment  Specialists 
^  Telecommunication  Specialists 


Phase  2:  Workforce  Training 


S  Determine  appropriate  amount/level  of  training  by 
working  with: 

'S  Internal  engineering  management 
x  Center-level  functional  offices 
Higher  headquarters  (HQ  AFMC) 

'f  Senior  Program  Managers 

S  Ensure  training  consistent  with  other  ALC,  AFMC  and 
DoD  efforts 

S  Determine  training  plan  robust  enough  to  make  a 
difference  yet  realistic  based  on  workload 

□  Train  Workforce 


Phase  2:  Workforce  Training 


Courses  Selected  (All  CBT): 

•  SYS  182  Intro  to  Systems  Engineering  -  3  hrs 

•  SYS  155  Operational  Safety,  Suitability  &  Effectiveness  -  9  hrs 

•  SYS  028  Intro  to  Configuration  Management  ~  16  hrs 

•  SYS  165  Intro  to  Risk  Management  -  21  hrs 

•  SYS  172  Modification  Management  Process  -  6  hrs 


Phase  2:  Workforce  Training 


Org  A  Training  Progress  (45  People) 


100 

90 

80 

70 

Percentage  60 
Complete  50 

40 

30 

20 

10 

0 


••••  4th  Qtr  Goal 
••••  3rd  Qtr  Goal 


••••2nd  Qtr 


1st  Qtr 


Determine  Applicable  Programs  Criteria 


All  OSS&E  Programs 


C-9 

* 

C-12 

C-20 

* 

C-21 

C-21 

* 

C-26 

C-38 

* 

E-9 

KC-10 

* 

KDC-10 

Peace  Lotus 

* 

T-43 

Academy  Fleet 

* 

E-4B 

VC-25 

* 

HFGCS 

Ensure  processes  incorporate  systems 
engineering 

S Jumpstarted  SE  revamp  with  3  Key  Processes: 

Requirement  Management  01 

Risk  Management  01  _ 

S Test  Management  01 

Will  springboard  into  other 
organizational  processes 

□Goal  to  standardize  and  share 
Best  Practices  across  organization 


Phase  4:  Identify  and  Define  Processes 


Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Verification 

Loop 


Requirements 

Loop 

Functional 

Analysis/ 

Allocation 

Design 

Loop 


Design 

Synthesis 


Analysis  & 
Control 


OUTPUTS 


Phase  4:  Identify  and  Define  Processes 


•  Break  requirements  down  in  a 
Requirements  Correlation  Matrix  (RCM): 

•  Spreadsheet  with  following  columns: 

-  Requirement 

-  Requirement  Source 

-  Derived  Requirements 

-  Quantification 

-  Initial  Risk  Assessment 

-  Operational  Conditions 

•  Give  RCM  to 

-  Test  Team  for  their  planning 

-  Risk  Mngt  Team  for  their  planning 


Req  Req 
Title  Source 


Derived 

Req 


Req 


Op 


Risk 


Definition  Quantification  Cases  (R/Y/G) 


Phase  4:  Identify  and  Define  Processes 


Doc. 


g 


100%- 


□Ensure  Systems  Engineering  is  an 
incentivized  factor  in  all  applicable 
contracts 

S List  all  Contracts 

□Determine  which  should  have  SE 

□Determine  appropriate  SE  wording 

□Develop  plan,  so  as  contracts 
renewed,  incorporate  SE  incentivization 

□Work  with  contractor/partner  to  improve 
implementation  of  systems  engineering 


75% - 


50%  - 


25%  - 


Goal 


Contracts 


Already  incorporated  in  two  major  contracts! 


Phase  6:  Develop  Library  of  Tools 


□Need  good  SE  “toolbox” 

-  Templates 

-  Metrics 

-  How-to’s  (fishbone,  5-whys,  paredo, ...) 

-  Lessons  Learned 

-  Explanations 

-  Best  Practices 

-  Peer  Review 

-  Case  Studies 

-  Life  Cycle  Cost  consideration 

-  Contractual  language 

-  Etc... 


Functional  Office  to  Develop/Obtain. ...Not  Started  Yet 


Phase  7:  Track  Progress  via  Metrics 


S  Metrics  developed  to  track  progress 
S  Metrics  shown  regularly  to  upper  management 

□  1st  staff  meeting  of  month 

Quarterly  Weapon  System  Reviews 
S  Metrics  must  be  able  to  roll  up 

S  Metrics  will  track: 

Systems  Engineering  Implementation 
Requirements 
^  Risk 
S  Processes 
^  Training 
Contracts 


Phase  7:  Track  Progress  via  Metrics 


Phase 


Sys  Eng  Implementation  Progress 


CO  K) 

2  2  2  2  2  0  o  o  oooooooooooo 

t]  t]  t]  t]  t]  "n  -n-n  ~n~n~n~n~n~n~n~n~n~n~n~n 

-<-< 

OOOOOo  oo  oooooooooooo 

OlOlOlOlOo  O  O  'slSSSCOCOOOOO0000 


Still  Refining  Goal  Dates 


Sample  Program  Sys  Eng  “Dashboard” 


High 


Med. 


Low 


Requirements 


0/1 


Low 


Med. 

Risk 


2/3 


High 


Contracts 


Lean 


Doc. 


Training 


Processes 


Sample  Organization  Sys  Eng  “Dashboard” 


What’s  Next 


•  Continue  implementation  throughout 
organization 

•  Measure/Track  results 

•  Increase  and  standardize  systems 
engineering  presentation  in  quarterly 
Weapon  System  Reviews  and  Staff  Mtgs 

•  Start  configuration  management  Process 
Improvement  Team 


Systems  Engineering  can  be  implemented,  applied 

AND  make  a  difference 


Summary 


•  Recent  high-level  policy  issued 

•  727th  ACSG  developed  grass-roots  means  to 
implement  SE  in  Sustainment  Environment 

•  Clear-cut,  tangible  processes  steps  for  the 
working-level;  metrics  to  measure  progress  for 
management 


I  n  Place  and  I  n  Use  Now 


Wi 


Questions  ? 


Technical  Data 

(an  Output  of  Systems  Engineering) 
in  the  Context  of  the  LCMC 

9th  Annual  Systems  Engineering  Conference 

23-26  October  2006 
San  Diego,  CA 
R.  Miller 
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Today’s  Agenda 


Statement  of  Purpose 

Government's  Weapon  system  (sustainment)  mission 

Background 

Challenges 

Path  Ahead 

Overview  of  the  SE  Process  from  a  DOD  perspective 
Technical  Data  as  a  specific  SE  Output  and  Metric 
Life  Cycle  Management  Command  (LCMC)  Org. 
Electronic  Data  Transfer 

Product  Data  and  Engineering  Working  Group  (PEWG) 
Summary 


Statement  of  Purpose 


The  importance  of  technical  data  and  its 
benefits  within  the  Government  is 
resurfacing  with  the  revitalization  of 
Systems  Engineering  by  OSD 


The  Government’s  Weapon  System 
(sustainment)  Mission 


The  OSD/Army  missions  requiring  TD 

-  FAR/DFAR  compliant  contracting 

-  Contingency  sustainment  strategies 

-  Spare  part  technical  designs  and  reviews 

-  Rapid  Warfighter  and  or  Operational  solutions 

-  Continuous  technology  refreshment 

-  Support  to  the  AM  LCMC  depots 

-  Competition  initiatives 

-  Reliability  Improvements 

-  Value  Engineering 

-  Hosting  standardized  TD  formats  for  seamless  transfers 


Background 

Acquisition  Reform  1 

-  Contractor-maintained  design/configuration 

•  Use  of  technical  data  within  Government  deemphasized 

•  Technical  Data  access  preferred  over  delivery 

-  Decentralized  program  management 

-  Performance-based  development  heralded  as  the 
panacea 

-  Streamlined  ECP  approval 

-  Elimination  of  Mil  Specs  and  Mil  Standards 

-  Elimination  of  non-value  added  reporting 

-  Rights  in  technical  data  deemphasized 


Savings  expected,  but  not  yet  realized 1 


Challenges 


•  Understanding/Overcoming  execution  issues  of  the  past 

-  Practices  of  the  past  that  were  based  in  paper,  presented  significant  issues 

•  Understanding  that  development  costs  are  equivalent  to  Rights  in  Data  (RID)  costs  12 

-  Costs  to  acquire  TD  rights  are  incorrectly  viewed  as  duplicative 

•  Understanding  the  USC  and  FAR/DFAR  establishes  significant  RID  advantages  for  OSD 

-  DOD  has  claim  even  to  privately  funded  TD 

•  Understanding  there  are  no  free  lunches  in  TD  maintenance 

-  TD  left  with  an  OEM  offers  little  advantage  while  presenting  significant  sole-source  challenges 

-  Access  only;  TD  strategies  are  not  adequate  for  long-term  LCMC  missions 3 

•  Understanding  PBL  and  performance  strategies/outcomes  are  necessarily  complex 

-  PBL  contracts  being  awarded  require  technical  data  contingency  or  exit  strategies 2 

-  Success  is  nebulous  at  best  (GAO) 

-  PBL  strategies  confuse  roles  of  Army  and  contractor 13 

-  Performance  in  general  adds  cost/risk  due  to  complex  preparation  1 

•  Decentralized  program  management  adds  cost/risk: 

-  Standardized  TD  delivery 

-  PMOs  must  support  collaborative  product  data  management 

-  Requires  core  logistics  capability  1 

-  Performance  Specifications  beneficial.... when  practical 1 
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Confluence  of  Solutions 


Today  with: 

1 .  The  revitalization  of  Systems  Engineering  is 
ushering  in  the  reemergence  and 

importance  of  technical  data 

2.  Meaningful  reorganizations  (Army  LCMC) 

3.  Electronic  data  transfer 

4.  Product  Data  Initiatives  by  Army  leadership  10 


A  suitable  path  ahead  is  beginning  to  take  shape 
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Systems  Engineering  a  DOD  Overview  11 


•  Requirements  Development 

-  Performance  Parameter  Objectives 

-  Affordability  constraints 

-  Scheduling  constraints 

-  Technical  constrains 

•  Logical  Analysis. . . .Functional  Analysis/allocation 

-  Partitions  a  system  in  to  logical  groupings 

-  Disciplined  definitions  of  interfaces 

•  Design  solution 

-  People  Products  and  process  entities 

-  Related  internal  and  external  interfaces 

-  Upward  and  downward  traceability 

-  Interoperability  and  open  system  requirements 
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Systems  Engineering  a  DOD  Overview  cont. 


•  Implementation 

-  System  elements  formed/fabricated 

-  Some  testing 

-  Packaging  handling  and  storage 

-  Operations,  maintenance,  installation 

•  Integration 

-  Interfaces 

-  Putting  it  all  together 

•  Verification 

-  Confirmation  that  the  system  meets  design  specifications 

•  Validation 

-  Confirmation  that  we  “..built  the  right  thing..” 

•  Transition 

-  Movement  of  the  system  to  the  ultimate  user 


Technical  data  performance  =  Program  performance 
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Technical  Data  Outputs  of  the  SE  Process 


Prelim  System  Spec 
Support/Maint  Concepts 


System  Performance  Spec 
System  Support/Maint.  Objectives 
Footprint  Reduction 
Acquisition  Strategy 


SRR,  SFR,  PDR,  CDR 
Initial  Product  Baseline 
FCA 

Capability  Prod.  Doc.  (CPD) 


Production  Baseline  (PCA) 


Technical  Data  as  an  SE  Metric 


LCMC  Re-Organization 


Impact  from  PMO  decisions  rested  with  those  outside  the  PMO 


LCMC  Re-Organization 


Impact  of  PMO  decisions  rest  in  the  PMO 
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Electronic  Data  Transfer 


•  Technical  data  today  can  be  moved 
electronically 

•  Technical  data  today  can  be  audited 
electronically 

•  A  “Government”  copy  of  the  TDP  no  longer 
required 

•  Lean  Six-sigma  and  good  SE  practices  ‘should’ 
be  driving  maintenance  (ECP)  costs  lower 

•  Model  based  Definitions  becoming  better 
defined/standardized 


Legacy  paper-based  issues  are  no  longer  relevant 


PEWG  Focus 


Philosophy:  Getting  Back  To  Basics 

-  Army  manages  Army  weapon  programs 

-  Supports  OSD  revitalization  of  Systems  Engineering 

-  Compliments  Acquisition  Reform 

-  PD  in  the  Army  is  essential  going  into  the  future 


Task  Description: 

-  Review  and  correct  existing  acquisition  policy,  instructional 
material  and  contracting  package  templates,  to  insure  the  Army 
contracts  for  and  receives  weapon  system  product  data  and 
appropriate  rights  to  use  that  data  in  a  knowledge-based  future 

of  organic,  contracted  and  contingency  life  cycle  support. 


Getting  back  to  the  basics 
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PEWG  Initiatives 


Mandate  the  delivery  of  PD  to  the  Army  in  all  development  and  S&T 
programs  through  clear  and  concise  direction 


Insure  the  Army’s  intellectual  property  rights  to  PD,  are  identified, 
correctly/consistently  marked  and  challenged  (if  necessary) 

Insure  that  DAU  reinforces  the  importance  of  PD  and  clearly  identifies  the 
potential  benefits,  legitimate  costs  and  uses  of  PD  when  instructing  PMs 
and  acquisition  professionals 

Stem  the  vast  diversity  in  Army  contracting  today  whose  PD  approaches 
range  from  abandonment  of  PD  to  the  delivery  of  the  full  technical  data 
package.... with  the  former  often  occurring  in  conjunction  with  PBL 
strategies 

Develop/adopt  mutually  acceptable  standardized  product  data  formats 


Getting  the  Government's  role  right 
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PEWG  Actions 


•  The  Defense  Acquisition  System  (DODD  5000.1) 

-  E 1 . 1 . 3  Competition  (ed  it) 

-  E 1 . 1 .4  Cost  and  Affordability  (ed it) 

-  E 1 . 1 . 1 6  Performance-Based  Acquisition  (ed it) 

-  E 1 . 1 . 1 7  Performance-Based  Logistics  (ed  it) 

-  El .  1 .29  Total  Systems  Approach  (edit) 

-  El .  1 .30  Intellectual  Property/Technical  Data  Rights 

•  Product  data  and  intellectual  property  rights  shall  be  vigorously  protected 

•  The  Defense  Acquisition  System  (DODI  5000.2) 

-  E10  ENCLOSURE  10 

•  Army  Acquisition  Strategy  (AR  70.1 ) 

-  Chapter  4  Acquisition  Strategy  (edit) 

-  Chapter  6;  Program  Design  (edit) 

-  Chapter  8;  Army-Unique  Policies  (new) 

•  Leverage  ASME  Y14.41,  MIL-T-3100C  standardized  formatting 


Adequate  policy  is  being  made  better.. more  practical 


Summary 


•  Technical  Data  is  an  essential  output  and  metric  of  the  SE  process 

•  LCMC  Reorganization  long  overdue 

-  PMs  are  not  always  right 

•  Preparation,  control,  management  and  distribution  of  technical  data 
can  be  conducted  electronically 

•  PEWG  is  correcting  our  course... getting  back  to  basics 

-  Acquisition  Reform  was  the  beginning,  adjustments  are  now  needed.. 

•  Define  (redefine)  our  mission/function 

-  Policy  is  interesting.... Execution  is  essential 

•  Industry  is  a  business  partner 

-  The  contract  and  PD  should  form  all  critical  communications 

-  There  are  no  free  lunches 

•  Performance  is  not  the  panacea 

-  Performance  is  still  transactional,  for  industry  or  Government 

-  Discreet/effective  performance  metrics  are  very  difficult  to  conceive 
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Definitions 


1.  PEWG:  Product  Data  and  Engineering  Working  Group 

2.  Army  Material  Command:  AMC,  Responsible  for  all  aspects  of  system 
sustainment 

3.  Functional  Centers:  Local  centers  of  support  with  a  specific  focus 

a.  AMRDEC:  Aviation  and  Missile  Research  Development  and  Engineering 
Center,  provides  all  aspects  of  technical  support,  SE,  Test,  CM,  Quality,  etc. 

b.  IMMC:  Integrated  Material  Management  Center  provides  all  aspects  of  supply 
and  inventory  management 

c.  AC:  Acquisition  Center  supports  all  aspects  of  the  formal  contracting  process 

4.  MSC:  Major  Support  Center  provides  dedicated  support  for  specific  commodities, 
aviation,  missile,  etc.  By  merging  collocated  acquisition  offices  (PMO)  with  the 
AMC  functional  organizations,  the  LCMC  is  formed 

5.  AAE:  Army  Acquisition  Executive  provides  component  (Army)  acquisition 
leadership 

6.  PEO:  Program  Executive  Office  provides  executive  oversight  of  assigned  project 
management  offices  (PMO)  within  a  specific  commodity 

7.  PMO:  Project  Management  Offices  provide  acquisition  support  to  develop  a 
capability  requirements  into  fielded  hardware  (materiel  developer). 

8.  Functional  Divisions:  Organized  groups  of  support  personnel  within  the  PMO 
organization.  Engineering  Group,  Logistics  Group,  etc. 
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Point  of  Contact 


Russell  F.  Miller,  russellT.miller@us.armv.mil,  256-876-1335 
Chief,  Technical  Data  Management  Division 
Engineering  Directorate 

Aviation  and  Missile  Research,  Development  and  Engineering  Center 
Huntsville,  AL  35898-5000 
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Missile  Operations  and  Support  R  ftkewi 

Simulation  (MOSS)  Method 


9th  Annual  NDIA  Systems  Engineering  Conference 

Rob  Slater 

Raytheon  Missile  Systems 
Tucson,  Arizona 
rdslater@raytheon.com 
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Missile  Operations  and  Support 
Simulation  (MOSS)  Method 

Modeling  Objective: 

Model  and  Analyze  Hardware  Stockpile  over 
Multiple  years  or  Program  Life-Cycle  to 
Predict  Repair,  Readiness,  Cost,  etc... 

■  Basic  Needs 

■  Solution  through  MOSS 

■  Follow-on  Work 
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Basic  Needs  to  Assess  Stockpile  m"*** 

Life-Cycle  Repair,  Readiness,  Cost,  etc... 
Account  for... 

•  Maintenance,  Testing,  Training,  Operational 
Tempos 

•  Dynamically  Changing  Utilization  of  Inventories 

•  Hardware  Reliability  in  Diverse  Environments 

•  Reliability  Growth  and  Wear-out 

•  Upgrade  /  Retrofit  Programs 

•  Effectiveness  of  Test  Equipment 

•  Expediency  of  Logistics  Supply  and  Transport 
Chain 

Factors  Interact  to  Affect 
Repair,  Readiness  and  Cost 
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Least  Abstract 

◄ - 


Actual  Experimentation 
On  Hardware  (Surveillance) 

Most  Expensive 

t 


Abstraction 


l 


Computer 

Simulation 


Cost 


/ 


Most  Abstract 


Closed-form  Math  Modeling 
(Equations  and  Spreadsheets) 


Least  ^xpensive 


Better  and  Cheaper  Software  Technologies  are 
Driving  Simulation  Costs  Down 


Benefits 

Systems  Approach 
Track/Update  Items  through  Process 
Easily  Characterize  Random  Variables 
Easy  to  Capture  System  Dynamics 
Easy  to  Characterize  Complicated 
Process  Flows 


Cons 

•  Non-Repeatable  Model  Build 

•  Difficult  Validation  &  Verification 

•  Still  Relatively  High  Cost 
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Logistics  Modeling  and  Simulation  with  MOSS 


MOSS  :  Raytheon  Simulation-Based  Method  for 
Modeling  O&S  Processes  of  Military  Fielded  Inventory 


Purpose  -  Predictive  Analyses 

•  Readiness  of  Inventory  (Stored  &  Deployed) 

•  Estimate  O&S  Cost 

•  Logistics  Pipeline  Capacity  Requirements 

•  High  Fidelity  Estimate  of  Depot  Returns  over  System 
Life-Cycle  for  Maintenance  Planning  and  Warranty 
Analysis 
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Elements  of  MOSS 

Common  Attributes,  Common  Blocks,  and  Sub-Models 

-  Common  Attributes  :  Characteristics  of  Missile-Items  that  are 
Prevalent  for  Most  Missile  O&S  Systems 

-  Common  Blocks:  Provide  Functionality  that  is  Prevalent  in  Most  O&S 
Systems.  Stored  in  Libraries 

-  Sub-Models:  General  Arrangements  of  Common  Blocks  that  provide 
Higher  Level,  More  Complex  Functionality 

Common  Blocks,  Attributes  and  Sub-models  provide  Pre-Validated 
Mathematics,  Are  Re-Usable  and  Streamline  O&S  Modeling 

Also  Promote  Model  Repeatability 
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Common  Blocks  of  MOSS 

To  Help  Define  Static  Logistics  Network 


.r  ;:7\  V 


HI 


Set  Failure 
Variates 


Common  Blocks 


FAim 


BIT  Test 


passL-I 


Other  Parts  of  Model 
accumulate  Duty  Cycles^ 


JNO 


JZ|  Warranty 
Check 

outO 


Common  Blocks  Contain  Pre-Validated  Logic  and  Math ,  and  They  Are 

Stored  in  Libraries  to  Promote  Re-Use 


11/1/2006  Page  7 


MOSS  Modeled  System 
An  Example 


MOSS  Broad-Block  Approach 


Model  the  Static  Life  Cycle  Network  from  Flow  Diagram 
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MOSS  Broad-Block  Approach  ->  Apply  Sub  Models 
(Arranged  Common  Blocks)  for  Specific  Functionality 


Sub-Model  for  Stockpile  Availability  Applied 
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MOSS  Broad-Block  Approach  ->  Apply  Sub  Models 
(Arranged  Common  Blocks)  for  Specific  Functionality 


Now  Apply  Sub-Model  for  Operational  Availability 
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MOSS  Method  Addresses  Needs 


1) 

• 

• 

2) 

3) 

4) 

5) 

6) 

Larger-Scope  System  Approach  Compared  to  Spreadsheets  and  Equations 

Can  Track  and  Update  Items  as  They  Move  through  Process 

Can  Easily  Characterize  Random  Variables 

Easy  to  Capture  Dynamic  State  Changes 

Easy  to  Characterize  Complicated  Process  Flows 

Standardized  Logistics  Tool  Set  in  Pull-Down  Menu 

Tools  are  Pre-validated 

Tool  Set  Induces  Repeatable  Structure,  Level  of  Detail,  and  Speed  of  Creation 
for  Future  Models 

Maintenance,  Testing,  Training,  Operational  Tempos 

1)2)  3) 

Dynamically  Changing  Utilization  of  Inventories 

2)  3) 

N  Hardware  Reliability  in  Diverse  Environments 

1)3) 

p  Reliability  Growth  and  Wear-out 

1)2) 

l_  J 

i-  Upgrade  /  Retrofit  Programs 

1)2) 

L 

Effectiveness  of  Test  Eguipment 

3)  4) 

n 

Expediency  of  Logistics  Supply  and  Transport  Chain 

1) 

Non-Repeatable  Model  Build 

4)  6) 

Validation  &  Verification 

5) 

Cost 

4)  5)  6) 
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Benefits  of  MOSS  -  Summary 

“Environmental  Fidelity”  equates  to  High  Fidelity  Prediction 

Tracks  and  Accumulates  Time  Spent  in  Various  Environments  for  Each  Item  in 
the  Inventory.  Accurate  Estimate  of  Duty  Cycle  ->  Accurate  Failure  Prediction 

Integration  of  Analyses 

Sub  Models  for  Failure  Prediction,  Warranty  Failures,  Availability  Analysis,  Spares 
and  More  ->  Consistency  Between  Different  Studies  in  the  O&S  Arena. 

Re-usable  and  Repeatable 

Common  Blocks  are  Pre-defined,  Pre-validated  and  Stored  in  Pull-Down  Library. 
Attributes  and  Sub  Models  are  Pre-defined 

“Transparent”  Interface 

MOSS  Models  are  Designed  For  Making  the  System  more  Understandable 
Through  use  of  Time-based  Statistics  and  Charts,  Graphics,  Hierarchy  and 
Animation. 

The  Act  of  Building  Models  with  the  MOSS  Method  Aids  and 
Promotes  Model  Verification  and  Validation 
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Follow-On  Work:  Enable  Direct  Life-Cycle  / 
Mission  Trade  Analyses 


DESIGN  Inherent  Reliability  —  Service  Life  - 
Maintainability  and  Testability 


LOGISTICS  Maintenance  -  Upgrades  -  Training  - 
Pipeline  Infrastructure  -  Forward  Supply  &  Readiness 


Performance  Driven 
Asset  Demand 


PERFORMANCE  End-Game  Mission  Effectiveness 
and  Holistic  Cost  for  Mission  Success 


Benefit:  “Cradle-to-Target”  Insight  into  Performance  &  Cost 
Shortfall:  Vertical  Chain.  Real  World  can  be  Horizontal 
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Follow-On  Work:  Integrate  Logistics  iw** 

Modeling  with  Other  Initiatives 

Integrate  Logistics  /  Mission  Performance  Modeling  and 
Simulation 

Example  :  Total  Asset  Visibility  and  Prognostics  could  Update 
Predictive  Models  with  Actual  Field  Data 


11/1/2006  Page  15 


Suitable  Metrics 
for  Measuring  the 
Effectiveness  of 
the  Systems 
Engineering 
Process 


Benjamin  Q.  Luong 
and 

Lauren  L.  Nguyen 
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•  As  we  strive  to  develop  resilient  large-scale  integrated  products  and  enterprise  systems 
that  are  entirely  inventive  and/or  significantly  better  than  their  predecessors,  the  Systems 
Engineering  discipline  leads  us  to  improve  product  definition,  product  development,  and 
overall  performance  of  businesses,  particularly  commercial  and  defense  programs. 

•  In  recent  years,  Boeing  has  made  significant  leaps  in  improvement  to  the  application  of  the 
Systems  Engineering  Process,  especially  in  the  areas  of  requirements  management,  risk 
management,  trade  studies,  and  verification  and  validation.  During  our  continuous 
improvement  journey,  there  is  a  need  for  a  way  to  measure  the  effectiveness  of  Systems 
Engineering  on  program  performance.  This  presentation  describes  a  set  of  Systems 
Engineering  ‘program-level’  predictive  and  reactive  metrics  that  give  clear  visibility  into  the 
benefit  of  Systems  Engineering  on  program  performance. 

•  Further,  these  program-level  metrics  are  also  flags  that  help  forecast  positive  or  negative 
events.  Each  predictive  metric  has  a  high  or  low  correlation  counterpart(s),  its  reactive 
metric(s).  As  we  determine  the  relationship  between  predictive  and  reactive  metrics,  they 
can  be  mapped  to  different  pinpoints  in  the  Systems  Engineering  Process.  Also,  a  reactive 
metric  within  the  process  may  be  predictive,  which  may  map  to  another  reactive  metric.  By 
understanding  and  evaluating  the  results  of  these  metrics,  we  can  continuously  improve 
the  application  of  Systems  Engineering  and  ultimately,  program  performance. 
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•  Systems  Engineering  Process  Fundamentals 

•  Thinking  about  Effectiveness 

•  Thinking  about  Performance 

•  Measures  of  Effectiveness  &  Performance 

•  Suitable  Measures  of  Effectiveness  &  Performance 
for  Systems  Engineering  Process 

•  Suitable  Systems  Engineering  Index 

•  Predictive  and  Reactive  Metrics:  Examples 

•  Predictive  and  Reactive  Relationships 

•  Corporate  /  Program  Performance 

•  Conclusion 
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Systems  Engineering  Process  Fundamentals 


•  Process  input 

•  Customer  needs/ 
objectives/ 
requirements 

-  Missions 

-  Measures  of 
Effectiveness 

-  Environments 

-  Constraints 

•  Technology  base 

•  Outputs  from  prior  phase 

•  Program  decision 
requirements 

•  Requirements  applied  through 
specifications 

and  standards 


Requirements  analysis 

•  Analyze  missions  and  environments 

•  Identify  functional  requirements 

•  Define/refine  performance  and  design 
constraint  requirements 


Requirements  loop 


System 
analysis 
and  control 
(balance) 


t 


Control 

loop 


\ 


Functional  analysis/allocation 

•  Decompose  to  lower-level  functions 

•  Allocate  performance  and  other  limiting  requirements  to  all 
functional  levels 

•  Define/refine  functional  levels 

•  Define/refine  functional  interfaces  (internal/external) 

•  Define/refine/integrate  functional  architecture 


| 

Systems  Engineering 
Management 

Risk  management 
Configuration  management 
Interface  management 
Data  management 
Performance  based 
progress  measurement 

-  IMP/IMS 

-  TPM 

-  Technical  reviews 


Design  loop 
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Verification  Loop 


Synthesis 

•  Transform  architectures  (functional  to  physical) 

•  Define  alternative  system  concepts,  configuration  items 
and  system  elements 

•  Define/refine  physical  interfaces  (internal/external) 

•  Select  preferred  product  and  process  solutions 

J 


Process  output 

•  Balanced  Product 

•  Phase  dependent 

-  Decision  support  data 

Copyright  ©  Boeing  2006  “  System  architecture 

-  Specifications  and  baselines 
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•  Take  off  your  Engineering  Cap 

•  Put  on  your  Business  Cap 

•  Ask  the  Fundamental  Business  Questions: 

-  What  does  effectiveness  really  mean? 

-  What  are  everyday  examples  or  types  of  effectiveness? 

-  How  do  we  measure  effectiveness? 
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•  Definition  of  Effectiveness 

-Adequate  to  accomplish  a  purpose;  producing  the  intended  result  or 
expected  result 

-  Descriptions:  Influence,  Efficiency,  Capability 

•  Various  Types  of  Effectiveness 

-  Corporate  Effectiveness 

-  Organizational  Effectiveness 

-  Product  Effectiveness 

-  Individual  Effectiveness 


10/26/2006 


Copyright  ©  Boeing  2006 


6 


•  Measures  of  Effectiveness  (MOE) 

-  Corporate  Effectiveness 

-  Organizational  Effectiveness 

-  Product  Effectiveness 

-  Individual  Effectiveness 

-  SE  Process  Effectiveness 


Performance! 

Performance! 

Performance! 

Performance! 

Performance! 


•  What  are  the  common  Measures  of  Performance  (MOP)? 
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Thinking  About  Performance 
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Measures  of  Effectiveness  &  Performance 
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•  Now,  put  your  Engineering  Cap  back  on 

•  Measures  of  Effectiveness  ‘for’  the  SE  Process 

1 .  Balanced  &  Validated  Product 

2.  Product  Operational  Safety,  Suitability  &  Effectiveness 
2.  Return  on  Investment  (ROI) 

•  Measures  of  Performance  ‘from’  SE  Process 

-  Apply  the  business  fundamentals  to  each  core  SE  Process  Element 

•  Requirements  Analysis 

•  Functional  Definition 

•  Design  Synthesis 

•  System  Analysis  &  Control 

•  Product  Verification  &  Validation 

-  Other  SE  Elements 

•  Suppliers  &  Supplier  Management 

•  Production 

•  Operations  Support 

•  Sustainment 
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Suitable  Measures  of  Effectiveness  &  Performance  for  SE  Process 
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•  Comprised  of  Several,  Weighted  Components  with  Trend  Curves 

-  Requirements  Analysis  Performance 

-  Functional  Definition  Performance 

-  Design  Synthesis  Performance 

-  System  Analysis  &  Control  Performance 

-  Product  Verification  &  Validation  Performance 

-  Suppliers  &  Supplier  Management  Performance 

-  Production  Performance 

11760 

-  Operations  Support  Performance 

-  Sustainment  Performance 

11700 

11680 

11660 
30 

2  20 
5  10 

EI 

0 
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•  Requirements 

-  Requirements  Quality 


Predictive 


•  Design  Synthesis 

-  On-Time  Engineering  Release  Reactive  -  Predictive 

-  After-Initial-Release  (AIR)  Traffic  Reactive  -  Predictive 


•  System  Analysis  &  Control 

-  Design  Reviews:  Number  of  Critical  Action  Items  Reactive  -  Predictive 

-  Risk  Management:  Number  of  Risks  Identified,  Reactive  -  Predictive 

Mitigated,  Retired,  Realized  &  Elevated 

•  Product  Verification  &  Validation 

-  Requirements  Compliance  Reactive  -  Predictive 


•  Production 

-  LRU  Tag  Trend  Reactive 

-  Deviations  &  Waivers  Reactive 
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•  Reactive  Indicators  of  Economic  Growth  (Common  Examples) 

-  Gross  Domestic  Product  (GDP) 

-  Stock  Market  Indices: 

•  Dow  Jones 
• NASDAQ 
•S&P  500 

•  Nikkei  225 

-  These  can  be  predictive  in  a  different  context  when  they  provide  leading  insight  into 
any  specific  economic  segment 

•  Predictive  Indicators  of  Economic  Growth  (Common  Examples) 

-  Inflation 

-  Manufacturing  &  Housing  Index 

-  Interest  Rates 

-  Employment  Rate 

-  Oil  Prices 

-  Corporate 
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Typical  Predictive  and  Reactive  Rel 

lationships  (snapshot) 

Predictive 

Reactive 


Metric  of 
Interest 


#  of  Critical 
Action  Items 


Requirements 
Quality 


One-to-One 


Many-to-Many 


o 


One-to-Many 

Requirements  Analysis 
System  Analysis  &  Control 
Design  Synthesis 

Verification  &  Validation 


#  of  High  Risks 
Identified 


After-Initial- 
Release  Traffic 


On-Time  Engr 
Release 


One-to-One 


After-Initial- 
Release  Traffic 


Many-to-One 


One-to-One 


Deviation 

Waiver 
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Higher-Level  MOEs 

-  Profit  Margins 

-  Cost  Reduction 

-  Return  on  Investment 

-  Customer  Satisfaction 

-  Customer  Confidence 

-  Program  Mgmt  Indicators 


Financial  Performance. 
Financial  Performance. 
Financial  Performance. 
Financial  Performance. 
Financial  Performance. 
Financial  Performance. 


•  Higher-Level  MOPs 

-  Quality 

-  Cost 

-  Schedule 

-  Productivity 

-  Resources 


Malcolm  Baldridge,  ISO,  CMMI  Level  5 
Lean,  Six  Sigma,  Savings,  Rework 
Timely  Product  Delivery 
Employee  Satisfaction 
People,  Processes  &  Tools 
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•  Systems  Engineering  effectiveness  can  be  gauged  through  performance  of 
Systems  Engineering  elements 

•  Typical  performance  indicators  may  include  Quality,  Cost,  Schedule, 
Productivity,  and  Resources 

•  Suitable  metrics  can  be  derived  from  performance  indicators 

•  Metrics  can  be  Predictive,  Reactive  or  both 

•  Relationship  between  Predictive  and  Reactive  metrics  can  be  correlated  to 
specific  elements/phase  of  the  Systems  Engineering  Process 

•  Continuous  evaluation  of  these  metrics  increases  Systems  Engineering 
effectiveness  and  overall  Program  performance 
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•  Benjamin  Q.  Luong,  Systems  Engineer 

-  Phone:  (562)  593-4668 

-  Email:  beniamin.q.luonq@boeinq.com 

•  Lauren  L.  Nguyen,  Systems  Engineer 

-  Phone:  (562)  593-4489 

-  Email:  lauren.l.nquven@boeinq.com 


Thank  you! 


Questions?  We  might  have  answers... 
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PBL  Overview,  Business  and 
Metric  Discussion 


Dr.  David  Nowicki  &  Dr.  Dinesh  Verma 
Stevens  Institute  of  Technology 

Mr.  Tom  Parry 

Decisive  Analytics  Corporation 


Agenda 


•  General  Insights  and  Findings 

•  PBL  Structure 

•  Literature  Results 

•  PBL  Analytical  Framework 

•  Performance  Metrics 

•  Future  Research 
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General  Insights 


PBL  Intent 


Satisfy  warfighter  requirements 


Create  an  optimized  weapons  system  support  infrastructure  which 
capitalizes  on  the  best  of  government  and  industry  capabilities 


Total  weapons  system  support  structure  designed  to  meet  end 
user  needs 


•  Utilizes  the  best  of  both  contractor  and  government  capabilities  to 
achieve  support  objective 

•  Purchase  total  lifecycle  support  versus  the  components  of  support 

•  Implements  best  commercial  practices  associated  with  supply 
chain  management 

•  Spreads  risk  traditionally  born  exclusively  by  the  government 

•  Reduces  the  transactional  intensity  associated  with  a  traditional 
support  system 

•  Leverage  commercial  investment  in  technologies  and  industry  best 
practices 
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Is  it  Really  PBL? 

•  Not  every  contract  with  “performance  based”  written  into  it  is  a 
PBL  contract 

•  Performance  based  contracting  in  place  for  years 

•  Value  based  contract  awards  based  on  factors  other  than  cost 

•  Performance  Metrics 

•  Incentive  fees  and  penalties 

•  True  PBL  is  much  more  expansive 

•  Fixed-price-per-unit  of  output  (i.e.,  flight  hour,  cycles  of  operation); 
efficiency  assured 

•  Fixed-price-per-unit-per-period 

•  Higher  profits  because  contractors  share  risk 

•  Incentive  to  continuously  improve  product  reliability  (effectiveness) 

•  Improve  capabilities  through  system  modernization 
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Commercial  Precedence  to  PBL 


PBL  has  been  in  use  by  Commercial  Industry  for 
over  20  years  and  is  emerging  as  an  Industry  best 
practice. 

Aircraft  Industry  has  led  the  way  with  concepts  such 
as  Power  by-the-hour  (unit  of  output) 

•  GE  -  Jet  Engines 

•  UT  Jet  Engines 

•  Pratt  Whitney  -  Jet  Engines 

•  Rolls  Royce  -  Jet  Engines 

•  Honeywell  -  Avionics 

•  Rockwell  Collins  -  Avionics 

•  Lucas  Aerospace  -  Landing  Gears 

•  Garrett  -  Auxiliary  Power  Units 
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Using  PBL 

20+  at  Lockheed  Martin 

Common  Ground  Station 

F-117 

TOW  HAS 

T-45 

JSTARS 

Shadow  Tactical  UAV 
NAVI  CP:  Aircraft  Tires 
NAVICP:  APU/TLS 
F-18 
DDX 

Deep  Water 

JSF 

CH-47 

AAAV 

FCS 


Defense  Programs 


F/A-1 8E/F 
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PBL:  Difference  from 
Traditional  Logistics  Support 

Traditional  Log  Support 

•  Time  and  material  contracts 

•  Contractor  paid  as  service  is  delivered  regardless  of  impact  on 
warfighter 

•  Government  owns  all  of  the  performance  risk 

•  Under  defined  or  lack  of  defined  scope 

•  No  investment  by  contractor  beyond  that  paid  for  by  government 

•  Government  sunk  cost  in  materials 

•  Government  owns  the  results  if  they  accept  the  product  or 
service 

•  Contractor  gets  paid  for  correcting  deficiencies  he  may  have 
created 

•  Government  responsible  for  mitigating  obsolescence  issues 
No  incentive  to  introduce  improvements 
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PBL:  Difference  from 
Traditional  Logistics  Support 


•  PBL  represents  a  state  change 

•  Focuses  directly  on  meeting  warfighter  defined  goals 

Shifts  weapons  system  lifecycle  sustainment  responsibility  to  the  PM 

•  Payment  based  on  results  not  delivery 

•  Fixed  price  per  unit  of  output 

•  Performance  Metrics  driven  incentives  and  penalties 

•  Long  term  contracts 

•  Contractor  profits  based  on  level  of  risk  sharing 

Implicit  assumption  that  the  contractor  will  invest  in  infrastructure  and  inventory 

•  Freedom  to  execute  the  work  the  most  efficient  manner 

Oversight  based  on  performance  metric  results  rather  than  inspection,  cost  and  pricing 
data  certification,  etc. 

•  Incentive  to  improve  reliability  to  lower  operating  costs 

•  Incentive  to  upgrade  to  maintain  product  viability 

•  Simplifies  financial  transactions 

PROFITS  RESULT  FROM  AVOIDING  COST  AND  DOING  THE  RIGHT  THING 
but . 


PBL  Literature  Review 


PBL  Literature 


•  Most  of  the  research  on  PBL  is  dedicated  to 
developing  definitions  and  framework  for 
implementation  of  PBL. 

•  Just  two  refereed  publications  in  scholarly 
journals. 

•  Most  of  the  research  are  in  the  form  of  in¬ 
company  reports. 
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What  Exists 


•  Framework  for  Implementation  of  PBL. 

•  Performance  Metrics 
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What  is  Missing 


•  Strategies  for  operationalization  of  PBL. 

•  Models  for  evaluation  of  PBL  metrics 

•  Optimization  models  for  management  of 
various  asset  classes  under  PBL 

•  Obsolescence  Management 

•  Risk  Analysis 
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Publications  in  Refereed  Journal 

•  Berkowitz,  D.,  Gupta,  J  N  D.,  Simpson,  J  T  and 
Mcwilliams,  J  B.,  ‘Defining  and  Implementing 
Performance  Based  Logistics,  Defence 
Acquition  Review  Journal,  255-267,  2005. 

•  Cunic,  B.  Performance  Based  Contracting, 
Hydrocarbon  Processing,  December  2003,  43- 
46. 
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Literature  outside  Public 

Sector 

•  Buyer-Supplier  relationships  have  been 
studied  extensively  in  the  marketing  channels 
and  supply  chain  management  literature. 

•  Even  in  supply  chain  management  literature, 
little  research  has  considered  the 
performance  outcomes. 
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PBL  Structure 


PBL  Key  Findings 


Author 

Domain 

Key  Findings 

Be  rends 

Oil,  Chemical 

(Netherlands) 

•  Incentives  for  demonstrated  performance  should  be  attractive  to  the 
contractor  over  the  entire  contract  duration. 

•  Critical  staff  on  both  parties  should  have  continuity  over  contract  duration. 

Bi  llinton 

Utilities  (Canada) 

•  System  performance  is  to  be  evaluated  against  a  performance  baseline  based 
on  historical  data. 

Bi  llinton. 

Pan 

Utilities  (Canada) 

•  Continuous  risk  assessments  should  be  carried  out,  based  on  actual  data. 

Brown , 

Burke 

Utilities  (US) 

•  The  payment  scheme  should  differentiate  between  a  dead,  reward  and  penalty 
zones,  delimited  by  upper  and  lower  performance  thresholds. 

•  Clients  can  mitigate  their  risks  through  the  adequate  definition  of  the 
geometry  of  the  mentioned  zones. 

Cunic 

Chemical  sector 

•  Essential  or  strategic  aspects  of  performance  need  to  be  identified,  and  ad- 
hoc  contracts  should  be  arranged. 

Fearnley, 
Bekken , 
Norheim 

Public 

transportation 

(Norway) 

•  Historical  performance  data  are  essential  in  setting  sound  objectives  in  PBLS 
contracts 

•  Clear  delimitation  of  responsibilities  of  client  and  contractor  will  diminish 
likelihood  of  eventual  disputes. 

•  Performance  guarantees  or  bonds  are  recommended  to  cover  potential 
difficulties  for  clients  in  materializing  penalties  imposed  on  contractors  for 
system  under- performance. 

Gilbertson 

Defense 

•  Ad-hoc  warranties  should  be  set  for  essential  system  performance 

requi  rements. 

•  Objectives  of  operational  performance  are  to  be  set  with  regards  to  a 
reference  point. 

Hensher, 

Houghton 

Public 

transportation 

(Australia) 

•  System  utilization  profiles  need  to  be  adequately  defined. 

•  Communications  and  data  exchanged  between  client  and  contractor  should  be 
transparent. 

•  As  important  as  the  transition  or  migration  to  a  PBSL  scheme  is  the  definition 
of  a  post-transition  growth  strategy. 

Rogers 

Communications 

(US) 

•  Application  of  a  PBLS  strategy  from  the  early  stages  of  product  design  and 
development  can  lead,  by  controlling  the  dominating  design  parameters,  to 
significant  reductions  in  life-cycle  costs. 

Smith 

Defense  (US) 

•  Performance  goals  and  schedule  for  achieving  them  are  the  main  elements  of  a 
PBLS  contract. 

•  A  plan  should  be  set  for  the  refurbishment  of  critical  components  over  the 
useful  life  of  the  system. 

Wasserman, 
Lam  be r son 

Defense  (US) 

•  A  reliability  growth  program  has  to  be  set  to  compensate  for  the  decline  over 
time  of  the  system  reliability  characteristics. 

(1  ) 

Defense  (Spain) 

•  Multiple  contracts,  not  always  duly  cross-referenced,  complicate  the  true 
determination  of  responsibilities  when  assessing  fulfilment  (or  not)  of  system 
performance  goals,  and  thus  consistency  and  coherence  of  all  inter-related 
contracts  is  a  must. 

(1  ) 

Public 

transportation 

(Spain) 

•  Appropriate  delimitation  of  responsibilities  is  essential,  as  failure  to  do  it  may 
mean  that  the  contractor  is  eventually  held  responsible  for  under- performance 
of  the  system  caused  in  part  for  reasons  beyond  his  control. 

(1)  Based  on  the  authors’  experience  and  the  interviews  conducted  as  part  of  this  research  effort- 
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Concept  of  Dead,  Bonus  and 

Penalty  Zones 

Maximum 
Bonus 


Dead  Zone 


Penalty 
Zone 

Maximum 
Penalty 


3-D  of  Dead,  Bonus  and 
Penalty  Zones 
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3-D  of  Dead,  Bonus  and 
Penalty  Zones 


moe2l 


moe2u 


High 

Penalty 

Zone 

Low  Penalty 

Zone 

Conflict 

Zone 

Low 
Penalty 
"  Zone 

Dead 

Zone 

Low  Reward 

Zone 

I _ I  : 

Conflict 

Zone 

Low  Reward 

Zone 

High  Reward 

Zone 
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Contract  Framework 


Three  Critical  Aspects 

•  Will  and  capability  of  entering  into  commitments 

•  Contract  Purpose 

•  Rewarding/Penalty  Scheme 

Additional  Recommendations 

•  Legal 

•  Policies 

•  Information 

•  Supportability 
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Will  and  Capability 


•  Will  of  both  customer  and  contractor  to  enter 
into  the  contract  must  be  explicitly  stated 

•  Both  sides  must  have  appropriate  authority 
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Contract  Purpose 


•  Must  clearly  define  the  system  for  which  support  is  sought 

•  Must  include 

•  Definition  of  boundaries 

•  Primary  external  interfaces 

•  Definition  of  primary  system  elements 

•  Hardware,  software,  human  actions,  activities 

•  Details  of  support  objectives 

•  Contract  exclusions 

•  System  operational  life  defined 

•  Mission  profiles  and  durations 

•  Measures  of  performance,  how  they  are  measured  including 
frequency  of  measurement 

•  Continuity  of  key  personnel 

•  Identify  key  focal  points 

mi 
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Legal 


Fuzzy  and/or  ambiguous  statements  and  clauses 
are  to  be  avoided. 

The  contract,  both  in  its  spirit  and  in  the  specific 
wording  of  its  clauses,  is  to  be  in  full  accordance 
with  all  applicable  laws  and  regulations. 

The  contract  is  to  be  fair  and  balanced,  avoiding 
being  one-sided  in  either  direction. 

The  contract  is  to  define  applicable  jurisdiction,  in 
case  of  litigation,  as  well  as  the  procedures  for 
arbitration 
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Policies 


regarding  confidentiality  and  the  non-disclosure 
to  third  parties  of  sensible  information,  without 
the  pertinent  approval  of  the  other  party 

The  contract  is  to  reflect  that  both  parties, 
customer  and  contractor,  commit  to  sharing  all 
the  needed  information  and  to  revealing  it  to 
each  other  with  transparency  and  objectivity. 

The  contract  is  to  identify  the  applicable 
language  for  exchange  of  communications  in  the 
program 
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Information 


The  contract  is  to  clearly  identify  the  inputs  required  from 
third  parties,  whether  they  are  information,  services  or 
products,  that  may  affect  system  performance  and  that  fall 
out  of  the  responsibility  domains  of  both  the  customer  and 
the  contractor. 

The  contract  is  to  define  the  scope,  frequency  and  details 
of  the  reports  to  be  submitted  by  the  contractor  to  the 
customer  relative  to  the  development  and  results  of  the 
reliability  growth  program,  to  be  agreed  upon  between 
customer  and  contractor. 
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Supportability 

The  contract  is  to  define  the  procedures  to  be  followed  by  the 
contractor  and  the  customer  in  order  to  reduce  to  the  minimum  the 
risk  of  components  obsolescence  and  consequently  its  negative 
effect  on  system  performance. 

The  contract  is  to  include  a  technology  refreshment  program  for  all 
system  COTS  elements. 

The  contract  is  to  define  the  scope  and  details  of  a  reliability  growth 
program  aimed  at  compensating  the  wear-out  or  negative  effect  of 
time  and  use  on  system  reliability. 

The  procedures  for  dealing  with  refurbishment  of  critical 
components  and  for  the  pro-active  replacement  of  marginal 
reliability  components  prior  to  failure  (conditional  maintenance)  are 
to  be  defined  and  agreed  upon  between  customer  and  contractor. 

The  contract  is  to  define  the  configuration  management  procedures, 
essential  for  the  effective  and  efficient  conduction  of  other  activities 
aimed  at  ensuring  system  performance. 
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PBL  Analytical  Models 

and  Metrics 


A  Framework  of  Analytical  Models  to  Support  Initiating,  Trackiip 
and  Evaluating  PBL  Contracts 


Single  Metric  to  Monitor  and  Evaluate  PBL  Contract  Performance 

-  Use  of  Bayesian  Networks 

-  Improve  existing  search  algorithms 

-  Develop  an  Assessment,  Monitoring  and  Diagnosis  (AMD)  tool  for  PBL 

Metrics 

-  Outcome  Based,  Assessment  Focusec 

-  Material  Availability  (Operational  Avail 

-  Material  Reliability  (Mission  Reliability 

-  Mean  Down  Time  (Logistics  Respons* 

-  Understand  the  Interrelationships  of  tf 

-  Without  Exception,  metrics  to  date  are 
or  averages. 

-  Need  to  consider  variability 

-  Use  statistical  learning  techni 
than  lagging  metrics 

i  on  Goals  and  Variance  from  Goals 
ability) 

) 

3  Time) 

le  performance  metrics 
i  focused  on  single  point  measurements 

ques  to  establish  leading  metrics  rather 

Multi-Objective 

Multi-Resource 

-  Decisions  made  simultaneously 
considering  multiple  objectives 

-  Objectives  depend  on  Milestone 
(design  decisions  versus  monitoring 
sustainment  performance) 

-  Objectives  depend  on  system  of 
system  versus  system,  versus 
subsystem,  versus  LRU,  etc. 

-  Decisions  on  one  resource  (quantity 
and  location)  are  made  considering 
the  system  impact  of  not  only  this 
resource  but  all  types  of  resources 
(labor,  facilities,  material, 
transportation,  information,  etc.). 

-  System  impact  here  is  measured  in 
terms  of  revenues  received  through 
performance  and  cost  incurred  to 
achieve  performance. 
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Performance  Metrics 


•  Material  Availability 

•  Material  Reliability 

•  Mean  Downtime 

•  Outcome  Based  Assessment  Focused  on 
Goals  and  Variances  from  Goals 
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Life  Cycle  Metrics 


A  A  (Program  A 


Concept  Technology  System  Development  Production  & 

Refinement  Development  &  Demonstration 

Deployment 


,  Concept 
A  Decision 


Design 

.  Readiness  .  q 

A  Revi6W  LRIP/IOT&E  Af 


FRP 

Decision 

LRIP/IOT&E  AReview 


Pre-Systems 

Acquisition 


Systems  Acquisition 


65-80%  of  the  Life  Cycle  Cost 

Operations  & 
Support 


Sustainment 


PRE-IOC  AND  POST  IOC  SUPPORTABILITY  ASSESSMENTS 


JCIDS-^KPP— SEP — ►DAES 


One  set  of  Metrics  throughout  the  System  Life  Cycle 


Emphasis  on  Materiel  Readiness 
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Materiel  Availability 


Materiel  Availability 


Number  of  End  Items  Operational* 

Total  Population  of  End  Items 


*  Operational  means  in  a  materiel  condition  such  that  the  end  item  is  capable  of 
performing  an  identified  mission. 
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Materiel  Reliability 


Materiel  Reliability  =  Mean  Time  Between  Failure 

measured  as  Total  Operating  Hours 

Total  Number  of  Failures 
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The  research  alluded  to  an 
absence  of  good  models  for  the 
modeling  of  multiple  resources 
simultaneously.  This  aspect  of 
the  research  (multi-resource 
optimization)  is  being  led  by  Dr. 

PBL  Contractual  Performance  Outcome  David  Nowicki 

Structure  Metrics/Models  A  N-Dimensional  R&P  Scheme 


Depend  upon 
coherent 
metrics 

■=> 


Initially,  the  research 
involved  studying  PBL 
contracts  in  the 
commercial  and  public 
sector  to  understand  their 
structure.  This  research 
resulted  in  the 
development  of  a  PBL 
contract  structure 
template.  The  focus  from 
this  point  forward  was  on 
the  reward  and  penalty 
aspect  of  Performance  or 
outcome  based  contracts. 


Rewards 
&  Penalties 


Rewards 
8t  Penalties 


Basis  for  an  n- 
Dimensional 
Reward  and 
Penalty  Schema 

<=> 


Given  the 

dependence  of  good 
performance  based 
contracts  on 
coherent  and  sound 
metrics,  the 
research  next 
involved  a 
considerable  survey 
and  review  of 
metrics  and  models. 


This  aspect  of 
the  research 
is  currently 
being 
completed 


The  research  alluded  to  an 
absence  of  good  models  for 
systems  with  degraded  modes 
of  operation.  This  research 
took  a  deep  dive  into  this 
aspect  of  system  reliability 
modeling.  The  research  was 
accepted  for  publication  in 
Reliability  Engineering  and 
System  Safety. 
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Rapidly  Defining  a 
Lean  CMMI 
Maturity  Level  3 
Process 


Zia  Tufail,  zia@hp.com,  301.233.4228 

Julie  Kellum,  Julie.Kellum@hp.com,  404.731.  52.63 

Tim  Olson-QIC,  Tim.Olson@qic-inc.com,  760.804.1405 
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Presentation  Objectives 


i  n  v  e  n  I 


Describe  CMMI  compliant  online  HP  process 

Outline  Components  for  a  cost  effective,  lean  process 


Describe  HP’s  improvement  objectives. 

Identify  problems  addressed  by  the  streamlined  CMMI  L3  HP 
process 


Describe  the  approach  that  was  used  to  achieve  CMMI 
Maturity  Level  3. 

Unique  techniques  for  defining  &  implementing  an  effective, 
lean  CMMI  L3  process  in  8  months 


Present  some  challenges  and  lessons  learned. 


Answer  any  questions. 


October  26,  2006 
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Outline 


HP  Overview 


HP  Improvement  Objectives 


HP  Approach 


Challenges  and  Lessons  Learne 


Questions  and  Answers 
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FY05  C&l  EAS/Federal  Initiatives  E3 

invent 

EAS  &  Public  Sector  Initiative  CM  Ml  Relation 


•  CMMI  L3  Achievement 

•  Expand  Application 
Services 

•  Increase  Pursuit 
Capabilities 

•  Delivery  Excellence 

•  Profit  Improvement 


Development  of  EAS  HPGM-AS 
methodology,  rollout  and 
assessment. 

L3  methodology  is  unifying  force  for 
consistent  standards  and  practices. 


CMMI  allows  us  to  pursue 
opportunities  we  would  not  get 
otherwise 

Better  estimation  and  marketing  of 
services 

Standardization  and  enforcement  of 
best  practices. 

Disciplining  delivery  execution  and 
preventing  margin  leakage  by 
fostering  reuse  and  minimizing  risk 


October  26,  2006 
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Existing 
Methods 
Revised  to 
Meet  CMMI 
Level  3 


Pilots 

Underway 


2004 

A  $1.5  Million  dollar,  8  month  project  started  to  revise  Success  Program  W 
meet  CMMI  level  3  and  improve  project  controls.  Some  current  and  new 
FY05  Federal  Contracts  at  risk  without  a  CMMI  level  3  compliant  process. 

•  Sept  CMMI  Level  3  Gap  Analysis  on  Current  Methodology 

•  Oct  -Nov  HPGM-AS  Methodology  Customizations  -  define  per 

Sharepoint  portals 

•  Dec  Pilots  Start : 

•PM  Orientations/Planning  Sessions  per  project 
•CMMI  &  Methodology  Classroom  Training 


2005 

Success  Program  Office  (SEPG  team)  and  Pilot  Projects  start  leveraging 
Success  Program  revised  with  the  new  CMMI  level  3  compliant  process 


Jan 

Feb 

Mar 

April 


Envisioning  &  Maintenance  Phase  Pilots  start 
Design  &  Maintenance  Phase  Pilots 
Build  &  Maintenance  Phase  Pilots 
Pilots  Continue  &  CMMI  Assessment  begins 

April  29  EAS  and  Public  Sector  Achieve  CMMI  level  3  maturity 


October  26,  2006 
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Components  of  the  HP  CMMI  L3  Process 


i  n  v  e  n  I 


Technology 

Sharepoint  based  portals: 
Success  Program  Office 
Success  Manager 
Success  Reviews 
HPGM-AS  Roadmap 
RADM  Virtualization 


Process 

SEI  CMMI  L3  framework 

HP  Global  Method  Application  Services 
(HPGM-AS  -  methodology) 

Success  Reviews  (PPQA  audit  process) 

Success  Manager  (Team  Collaboration 
process) 

Adaptive  Assets  (Knowledge  Capture  & 
Reuse  process) 


Success  Program  Office 
(SEPG  function) 

SQA  Coaches  (per  team) 

Program  Management 

Organization  &  Culture 


People 


Oct 
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Success  Program  Office 


Application  services 
(HPGM-AS) 

•  Delivery  Workflow  portal/process 

•  CMMI  level  3  compliant 

•  Phase  deliverables  &  Signoffs 

Success  Manager 

•  Team  Collaboration  web/process 

•  Pre-Loaded  HPGM-AS 

•  Customer  accessible 

Success  Reviews 

•  Process  Review  (PPQA)  portal/process 

•  Mentoring  Support  By  Field  PMs 

Adaptive  Assets 
Knowledgebase 

Process  and  Tech  Asset  KB  portal/process 

•  Reuse  tool 

Rapid  ADM 

•  Supports  HPGM-AS  Configuration  Mgt 

•  Virtualization  and  Software  Configuration 

•  Management  (SCM)  services 
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Second  Release  of  HP  Global  Method  for  Application  Services  which  is  a  level  3  CMMI  application  development: 
methodology  supporting  iterative,  waterfall  and  maintenance  lifecycles  in  HP's  Consulting  and  Integration  (CSd)  area 


HPGM-AS 
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Success  Reviews  (PPQA) 
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CMMI  Links 

■  SUEMIT  Success  Program 
Improvements  Here! 

■  Adaptive  Assets 

■  Success  Reviews 

■  CMMI  project  webs  (currently  US  PS 
only  -  special  clearance  required) 

■  HPGM-AS  Lifecycles  Overview 

■  SPO  Portal 

■  SPG  Lessons  Learned  Analysis 

■  HPGM-AS  Impact  Analysis  Tool 

■  SPO  Configuration  Management  Plan 

■  SPO  Training  Plan 

■  SPO  Improvement  Plan  (SIP) 

■  SPO  Quality  Policy 

■  SPO  Quality  Assurance  (QAP)  Plan 

■  CMMI  Scope  &  Design  Decisions 


Stcoess  Reviewer 
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Rapid  ADM  Components  and 
Features 


Project  Team  Portal 


Success  Manager 
Pfoject  Team  Porta1  ^ 

are  Configuration  Man*^ 
Complis*^^ 

^fetion  Development  ^o$0) 

2|ng  Model  (SASU 


fOyi 


'sion 


>mated 
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Environment,^} 

Manager 


Rapid  ADM  Development  Platform 


October  26,  2006 


i  v  e  n  t 


9 


Outline 


HP  Overview 

HP  Improvement  Objectives 
HP  Approach 
Challenges  and  Lessons  Learne 
Questions  and  Answers 
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Success  Program  Vision 

Using  a  sales  and  delivery  integrated  approach,  leverage 
existing,  as  well  as  grow  new  process  and  technical 
Adaptive  Assets  to  : 

Increase  ability  to  compete  in  Federal  bids  through  CMMI  Level  3 
compliance 

Win  more  business  through  improved  differentiation 

•  IBM  differentiation  point 

Increase  Success  of  Projects 

•  Synergistic  Teams 

•  Referencable  Clients 

•  On  Time 

•  In  Budget/Profitable 

•  Effective  HP  Global  Methods  Applied 
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Success  Program  4  What  &  Why 


m 


invent 


Collection  of  specific  process  and  technical  assets  used  in  pre-sales  for 
differentiation  and  in  delivery  to  increase  project  success 


CMMM  Maturity  Level  3  compliant  to  meet  Federal  bid  requirements 


Externally  accessible  outside  HP  Firewall  by  delivery  teams 


•  Major  Areas: 

Success  Path  HPGM-AS 
Success  Manager 
Success  Reviews 
Adaptive  Assets  KB 
Communities  &  Forums 


Best  of  breed  IP  from  HP 
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Outline 


HP  Overview 


HP  Improvement  Objectives 


HP  Approach 


Challenges  and  Lessons  Learne 


Questions  and  Answers 


m 


invert 
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HP  Approach 


i  n  v  e  n  I 


Leverage  existing  HP  processes  and  templates: 

-  Success  Program 

-HP  India  (CMMI  Maturity  Level  5) 

-  HP  Global  Methods  for  Application  Services  (HPGM- 
AS) 

-  HPGM  for  Project  Management  (HPGM-PM) 


Deploy  Tailored  SEI  IDEALSM  Model  and  reuse 
existing  processes  to  be  CMMI  compliant. 


Leverage  Best  of  breed  IP  from  HP 
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SEI  IDEALSM  Model 


m 


invert 


SEI  IDEALsm  Model  for 
Process  Improvement 


Leveraging 


Stimulus  for 
Improvement 

Set  Context 
&  Establish 
Sponsorship 

( Establish  ^ 
Improvement  / 
Infrastructure/ 

1 r  r-r 

,  Appraise  &  \ 

I  IT 

M  u  4  a  4  u  II 

Diagnosing 


Acting 


Establishing 


SM  IDEAL  is  a  service  mark  of  Carnegie  Mellon  University;  Slide  Adapted  from  SEI 


Tailored  SEI IDEALSM  Model  1 

Initiating:  Sponsor  sold  idea  to  HP  senior  management. 


Diagnosing:  HP  India  performed  a  mini-appraisal  (e.g., 
Class  B). 


Establishing:  CMMI  Consultant  established  high-level 
plan  and  HP  established  a  CMMI  Team. 


Acting:  Architected  HP  processes  to  be  CMMI 
compliant  in  2  months;  Skipped  piloting;  Trained  and 
implemented  experienced  projects. 


Success:  Performed  independent  SCAMPI  A. 
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HP’s  “Lean”  Process 


i  n  v  e  n  I 


HP  Success  Program  is  a  very  “lean”  CMMI 
compliant  process  (about  25%  of  the  size  of 
the  HP  India  process). 


The  process  is  completely  online,  and  uses 
Microsoft  SharePoint. 


HP’s  process  is  only  ~25web  pages  in  size. 


HP  incorporated  best  practices  in  process 
definition 

e.g.,  “Defining  Short,  Usable  Processes  and  Procedures”, 
CrossTalk,  Olson,  Timothy  G.,  June  2006. 
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HPGM-AS  CMMI  L3  Methodology  E3 

^  i  n  v  e  r  I 


2  Paths  depending  on  project 
profile 


Large  HPGM-AS  CMMI 
Implementation 

-  Small  HPGM-AS  CMMI 
Implementation 

Excluded  are  projects 
required  to  use  the 
customer’s  app  dev 
methodology  &  staff  aug 


No  existing  projects  asked 
to  convert  -  new  only 


Large  and  Small  Project 
implementations  differ 
mainly 

•  on  use  of  Success 
Reviews  and 
Deliverables  Required 


l 


Project  Planning  (PP) 


\  Project  Mgt  Control  (PMC)  |  "" 

LDecfe^on^Ana^sis^&^KolutionJDARQ 


PM 


Tailoring  (IPM) 


I 


Tonfjgu^io^management(CM^^] 


JT 


CM /Test  Lead 


Success  Reviews  (PPQA) 


Measurement  &  Analysis  (MA) 

Organizational  Process  Improvement 
_ fOPD,  OPR _ 

Organizational  training  (OT) 


Success  Reviewer 


SPO  Managers 
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IHPGM-AS  Artifact  Comparison 
Example 


Small/Low  Risk 

Large/High  Risk 

Success  Manager  site 

(Issues  List  &  Change  Req/PR  List) 

Success  Manager  site 

(Issues  List  &  Change  Req/PR  List) 

Project  Plan  &  Configuration  Mgt  Plan 

Project  Plan  &  Configuration  Mgt  Plan 

Schedule 

Schedule 

Mini-Spec 

(Contains  sections  from  all  of  these 

Bus  Req  Spec  (BRS) 

Systems  Req  Spec  (SRS) 

Systems  Architecture  (SA) 

Detail  Design  (DD) 

V&V  /Test  Plan 

Req  Traceability  Matrix  (RTM) 

Peer  Review  Log/Checklist 

Mini-Spec 

CM  Audit  Checklists 

Success  Manger,  Product 

Milestone  Review  Phase  Checklists 

Peer  Review  Log/Checklist 

-  Project  plan,  SRS,  SA,  V&V  Plan 

CM  Audit  Checklists 

Success  Manager,  Code,  Product 

Milestone  Review  Phase  Checklists 
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Outline  K3 

HP  Overview 

HP  Improvement  Objectives 
HP  Approach 

Challenges  and  Lessons  Learn 

Questions  and  Answers 
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Challenges 

•  CM  Ml  Resources  were 
reallocated  to  the  new 
high  priorities. 

•  Resistance  to  change 
when  people  have  gone 
through  several  changes 
without  tangible  results. 

•  Aggressive  schedule 

•  Unexpected  events  such 
as  team  health  and 
organizational  changes 
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Lessons  Learned 


i  n  v  e  n  I 


•  It  takes  time,  senior  management  support,  and  mentoring  to 
change  behavior 

•  Traditional  Classroom  training  not  enough  -  critical  to  have  a 
local  resource  to  mentor  team  on  methodology/CMMi  activities 
and  artifacts 

•  Success  Reviews  critical  for  monitoring  and  mentoring  on 
process  use 


•  Critical  to  use  centralized  team  collaboration  tool  for  process 
support  (i.e.  PPQA,  Peer  Reviews,  Lessons  Learned,  Issue 
Tracking/Project  Change  control,  etc) 


Lessons  Learned 


i  n  v  e  n  I 


•  Spend  ample  time  in  planning  phase  to  lock  down  the 
scope  of  the  project. 

•  Obtain  and  maintain  executive  sponsorship  to  keep  driving 
the  project  on  schedule. 

•  Define  Configuration  Management  early  in  the  process. 

•  Follow  the  Configuration  Management  process. 

•  There  is  no  substitute  for  one-on-one  mentoring  for  the  late 
adopters. 

•  Ensure  core  team  understands  the  entire  process. 

•  Ensure  the  process  is  lean  and  relevant. 


Lessons  Learned 


i  n  v  e  n  I 


•  Engage  stakeholders  throughout  the  process. 

•  Third  party  consultants  can  provide  objective 
feedback. 

•  Build  a  diverse  team  and  leverage  diversity. 

•  Ensure  CMMI  expert  resources  are  on  the  team. 

•  CMMI  really  does  require  continuous  process 


improvement. 


What  Went  Well 


i  n  v  e  n  I 


•  Teams  reap  benefits  of  Peer  Reviews,  and 
Milestone  Reviews 

•  Pilot  projects  implement  a  consistent  delivery 
approach 

•  Improved  metrics  to  track  trends  across  projects 

•  Integration  of  virtualization  through  CM  process 

•  Increased  value  to  customer  through  improved 
communications  thru  SPO: 

-  Success  Manager  Team  Webs 

-  Status  Reporting 

-  Automated  Issues  and  Change  Request  tracking 
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What  Went  Well 


i  n  v  e  n  I 


•  Expanded  Rollout  of  Success  program  to  -300 

•  Implemented  new  Success  Program  processes: 

-  HPGM-AS  Bid  Review  (Pre-sales) 

-  HPGM-AS  Quickstarts  (Pre-Delivery,  1  week)  ^  tailoring 
of  the  process  by  SEPG  with  PM  and  SA 

•  Implemented  scalable  training  to  replace 
classroom  training: 

-  PM  CMMI  Training  Track  (14  webinars) 

-  Engineering  CMMI  Training  Track  (9  webinars) 
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HP  Overview 


HP  Improvement  Objectives 


HP  Approach 


Challenges  and  Lessons  Learne 


Questions  and  Answers 
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Questions? 


October  26,  2006 


i  n  v  e  n  I 


NDIA  Systems  Engineering  Conference 


A  New  Strategy  for  Submarine 
Payload  Integration 


The  VIRGINIA  Multi-Mission  Payload  Module 


GENERAL  DYNAMICS 


Electric  Boat 


Approved  for  Public  Release  Per  SOS 
Ltr  5720/OODT;  2003  -  0839  2/5/04 
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ELECTRIC  BOAT 


Advanced 

Development 


Sea  Trials 
&  Test 


Life-Cycle  Support 


Premier  Resource  for  Submarine 
Design  and  Construction  Technology 


GENERAL  DYNAMICS 

Electric  Boat 
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k  ii  \j\  ii  i\\ 


October  25,  2006  2 


Electric  Boat 

Locations  and  Staffing  (As  of  October  7,  2006) 


GENERAL  DYNAMIC! 

Electric  Boat 
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•  CONNECTICUT 

Operations  and  Support 
Engineering 

•  RHODE  ISLAND 

Quonset  Point  Facility 
Newport  Engineering 

•  NEW  YORK 

Knolls  Atomic  Power  Lab 

•  GEORGIA 

Kings  Bay  Trident  Base 

•  WASHINGTON 

Bangor  Trident  Base 
Puget  Sound  NSY 

•  WASHINGTON,  D.C. 

•  VIRGINIA 

•  UK  /  Australia  /  Other 

TOTAL 


3,901 

3,456 

1,907 

53 

183 

125 

108 

250 

15 

365 

68 

10,431 


1960 


1970 


1980 


The  VIRGINIA  Class  Submarine  is  One  of 
the  Most  Complex  Systems  in  Production 


Electric  Boat:  Prime  contractor  and  lead  design  yard 

Awarded  10  ships  of  an  anticipated  30-ship  class  -  valued  at  $13B 

Lead  ship  SSN  774  VIRGINIA  delivered  October,  2004 


Ml 

Tank 

Boeing 

777 

VIRGINIA  C 
Submarii 

Weight  (T) 

65 

250 

7,800 

Length  (Ft.) 

25 

200 

377 

Number  Systems 

25 

40 

200 

Crew  Size 

4 

10 

113 

Patrol  Duration  (Hr.) 

24 

8-14 

2,000 

Number  of  Parts  to  Assemble 

14,000 

100,000 

1,000,000 

Assembly  Man-hours/Unit 

5,500 

50,000 

>10,000,000 

Production  Time  (Mo.) 

7.5 

14 

55 

Production  Rate  (Units/Yr.) 

600 

72 

0.5-3 

BENERAL  DYNAMIC! 

Electric  Boat 


Challenge:  Moving  From  Today  s  Capability 
to  Tomorrows  Mission  Needs 


Todays  Capability 


Tomorrows  Capability 


Battle  Group  Mine  Warfare  Anti-Ship  Surveillance  &  Strike  Special  Anti-Submarine 

Support  Warfare  Intelligence  Warfare  Warfare  Warfare 


Approved  for  Public  Release  Per  SOS 
Ltr  5720/OODT;  2005  -  0235  3/15/05 
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The  Concept  Was  to  Minimize  the  Impact 
to  the  Payload  and  the  Platform 


GENERAL  DYNAMICS 

Electric  Boat 


Concept 


¥ 


id 
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war- 


o 


CJ 
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Key  Technologies  Can  Enable  the  Concept 


Conceptual  Requirements 

Standardized  Interface 
External  Payload  Enabler 
High  Payload  Density 
Fast  Logistics 

Reloadable,  Recoverable, 
Reusable 

Adaptable  to  Many  Sizes 
and  Shapes 

Work  in  conjunction  with 
a  payload  capsule 


BENERAL  DYNAMICS 

Electric  Boat 


. . 


ULCiUQ 


Key  Technologies 

•  Composite 
Structure 

•  Compact 
Expendable  Hatch 

•  Shock  Mitigation 

•  Network-Based 
Control  System 

•  Wireless 
Communications 

•  Inductive  Power 
T  ransfer 
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Payloads  and  Sensors  Studies  Transitioned 
into  Risk  Reduction  Demonstrations 


ylo- 


PPQi 
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FJbxjdJb  Payload  JVJoduJ 


Dsirn 


The  Flexible  Payload  Module  is  moving  from 
Concept  Exploration  through  Risk  Reduction 


Phase  0 


Phase  1 


Concept 

Exploration 


System  Analysis 
Reqts.  Definition 
Conceptual  Design 
Technology  &  Risk 
Assessment 
Prelim.  Cost,  Sched.  & 
Perf.  Concept 


Program 
Definition  & 
Risk  Reduction 


Concept  Design  Update 
Sub-system  Tradeoffs 
Preliminary  Design 
Prototype  Testing 
Manufacturing  & 
Supportability 
Considerations 


Phase  2 


Engineering  & 
Manufacturing 
Development 


Detail  Design 
Development 
Risk  Management 
Development  Test  & 
Evaluation 

System  Integration,  Test 
&  Evaluation 
Manufacturing  Process 
Verification 


Phase  3 


Production, 
Fielding,  & 
Operational 
Support 

Production  Rate 
Verification 
Operational  Test  & 
Evaluation 
Deployment 
Operational  Support  & 
Upgrade 
Retirement 

Replacement  Planning 


GENERAL  DYNAMIC! 

Electric  Boat 
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Analyzed  for  Pressure 
and  Shock  Loads 

71  Robust  structure  in  any  ' 
configuration 

71  Shock  testing  validated  ■ 

the  analysis  P 

Built  at  Quonset  Point  ‘ 

71  Completed  in  23  days 

7i  Exothermic  reaction 
easily  accommodated 


Shock 
Test  Article 


PIV 


m 


in 


ir 
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GENERAL  DYNAMIC! 

Electric  Boat 
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Compact  Expendable  Hatch  Supports 
High  Payload  Density 


Combination  of  Steel  Shell 
and  Syntactic  Buoyancy 
Element 

Eliminates  Hinges  and 
Actuators  to  Open  and  Close 

Hatch  Release  Actuator 
Designed  to  Maximize 
Packing  Factor 

Floodable  Air  Volume  Allows 
Hatch  to  Scuttle  at  Desired 
Time 


GENERAL  DYNAMIC 

Electric  Boat 
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Shock  Mitigation  Pads/Launch  Rails 
Accommodate  Joint  Forces  Payloads 


Closed  cell  foam 
provides  the  mitigation 
for  lighter  payloads 

7i  Compresses  under  f 

pressure  to  release  | 

capsule  for  launch  | 

o 

71  Reduces  capsule  < 

accelerations  to 
10-30  g’s 

Easily  reconfigured  for 
the  next  payload 


10( 


GENERAL  DYNAMICS 

Electric  Boat 


1,800  lbs/in  Pad  Stiffness 


100 


500 
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Many  of  the  Key  Technologies  Serve  to 
Increase  Payload  Density 


Expendable  hatch  minimizes 
mechanical  systems 

Foam  filled  structure  allows 
closely  packed  tubes 

Shock  mitigation  pads 
reduce  the  accelerations 
and  maximize  packing 
factor 


Section 


of  FPM 


JULCiUQ 
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Electric  Boat 
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Network-Based  Control  System 
Supports  Rapid  Payload  Insertion 


Scalable,  Flexible 
Architecture 

Built  upon  COTS 
Technology 

Provide  intelligence 
onboard  FPM  platform 

71  Enables  “plug  and  fight” 

Shipboard  User  Interface 
uploads  FPM  configuration 


GENERAL  DYNAMICS 

Electric  Boat 
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LAN 


WLAN 


FPM  GUI 
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Commercial  Wireless  Networking 

for  the  Ship/FPM  Interface 


Leverages  Network 
Based  Architecture 

Commercial  Standard 
IEEE  802.11b  WLAN 

Capable  of  maintaining 
1 1  Mbps  over  small 
seawater  gap 


Attenuation  of 
2.4GHz  RF  in  Seawater 


Maintained 


1  1.5 

Antenna  Separation  (inches) 


GENERAL  DYNAMICS 

Electric  Boat 
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Inductive  Power  Transfer  was  Demonstrated 
Using  a  Split  Core  Transformer 


•  Demonstrated  transfer  of 
power  from  ship  to  FPM 

71 20  kW  ultimate  rating 
7i  2  kW  for  demo  FPM 


Environmentally  sealed 

FPM-to-Capsule  inductive 
coupler  is  under 
development 


Output  Voltage  Regulation 
as  a  Function  of  Gap 
with  a  200  Watt  Load 
and  a  120uF  PFC  Capacitor 


Gap  (inch) 
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These  Technologies  Were  Integrated 
Into  a  Demonstration  FPM 


Upper  Housing 


GENERAL  DYNAMIC: 

Electric  Boat 
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Shipboard 
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These  Technologies  Were  Integrated 
Into  a  Demonstration  FPM 
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Upper  Housing 


October  25,  2006  19 


The  Full  Size  FPM  Was  Successfully 
Demonstrated  at  Electric  Boat 


•  Employed  all  key 
technologies 

•  Demonstrated  full 
system  functionality 

•  Executed  an 
automatic  launch 
sequence 

•  Test-fit  into  a  05- 
sized  missile  tube 
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A  Second  Generation  Flexible  Payload  Module  Was 
Built  For  the  2004  Navy  Silent  Hammer  Experiment 


FPM  Concept  Refinement 

•  Full  Scale  Production 
Methods 

•  Network  Based 
Control 

•  Wireless  Technology 

•  Buoyant  Capsule 
Integration 


Flexible  Payload 
Module  (FPM) 

Under  Construction  in 
Shipping  Cradle 
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FLEXIBLE 
PAYLOAD 

MODULE  HI 

-- 1 


FPM  integrated  into  a 
missile  tube  on 
USS  Georgia 
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The  Flexible  Payload  Module  Provides  an 

Open  Architecture  -  System  Level  Solution 

To  Meet  Future  Mission  Needs 

•  Reduced  cycle  time  to  field  new  mission  capability 
by  minimizing  payload  specific  ship  modifications 

•  Decoupled  payload  development  cycle  from  the 
ship  development  cycle 

•  Maintains  critical  warfighting  performance 

•  Risk  reduction  on  critical  technologies  has  been 
accomplished 


Electric  Boat  Is  Moving  Forward  With  The  Payload  Module 
Concept  For  The  VIRGINIA  Class  Submarine 
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Headquarters  U.S.  Air  Force 

Integrity  -  Service  -  Excellence 


Nunn-McCurdys  Aren’t  Fun! 


Mr.  Gary  Payton 
Air  Force  Deputy  Under  Secretary 

for  Space  Programs 
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Agenda 


■  National  Security  Space  Enterprise  (NSSE) 
acquisition  programs  health  concerns  &  Nunn 
McCurdy  breach 

■  Role  of  systems  engineering 

■  “Back  to  Basics”  block  strategy 


Our  programs  are  only  as  good  as  our  people 
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Nunn-McCurdy  Breach  Defined 


■  Breach  definition 

■  “Infraction  or  violation  of  a  law,  obligation,  tie,  or 
standard”...  and 

■  “Broken,  ruptured,  or  torn  condition...” 

■  Nunn-McCurdy  breach  when  PAUC  or  APUC 
exceed  baseline  value  by  25% 

■  Two  1990’s  flagship  programs  experienced 
Nunn-McCurdy  breaches 


Integrity  -  Service  -  Excellence 
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Nunn-McCurdy  Law  Requires 


■  SecDef  must  address  four  questions: 

■  Is  the  program  essential  to  national  security? 

■  Are  there  alternatives? 

■  Are  new  cost  estimates  reasonable? 

■  Can  management  control  costs? 

■  SecDef  has  three  program  certification  options: 

■  Certify  “as  is”  with  updated  cost  &  schedule 

■  Certify  “as  restructured” 

■  “Not  Certify”  -  TERMINATE 


Integrity  -  Service  -  Excellence 
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Space  Breach  Programs  -  SBIRS 

■  Part  of  Total  System  Program  Responsibility 
(TSPR)  experiment 

■  SBIRS  breach  in  2001 

■  Restructured,  BUT... 

■  2005  breach  again 

■  IPT  -  restructure 

■  Management  changes 

■  Other  impacts 


Integrity  -  Service  -  Excellence 
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SBIRS  Breach  Root  Causes 


■  Systemic  shortfalls  in  systems  engineering: 

■  No  systems  engineering  master  plan 

■  No  consistent  requirements  management 

■  No  single/linked  integrated  master  schedule 

■  Systemic  industrial  base  issues  with: 

■  Business  practices 

■  Personnel 

■  Economics 

■  Industrial  capacity 


Integrity  -  Service  -  Excellence 
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Space  Breach  Programs  -  SBIRS 


Integrity  -  Service  -  Excellence 
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Space  Breach  Programs  -  NPOESS 


Integrity  -  Service  -  Excellence 
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Space  Breach  Programs  -  NPOESS 


■  NPOESS  strategy:  sacrifice  cost  and  schedule  to  maintain 
performance  with  overruns  paid  from  management  reserves, 
program  slips  and  accepting  increasing  risk 

■  When  NPOESS  Nunn-McCurdy  breach  announced 

■  PAUC  =  82%  APUC  =  202% 

■  NPOESS  Nunn-McCurdy  certification  strategy: 

■  IPP  +  restructure  (fewer  spacecraft  and  fewer  sensors) 

■  Government  and  contractor  management  structure  change 

■  NPOESS  PEO  established 

■  Chief  Systems  Engineer  position  established 


Integrity  -  Service  -  Excellence 
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Lessons  Learned 


■  Unbridled  optimism  regarding  cost,  schedule, 
performance,  and  risks  is  a  recipe  for  failure 

■  Lexington  paper  scenario 

■  Understated  costs  leads  to  lower  budget 

■  Leads  to  industry  bidding  price  less  than  budget 

■  Leads  to  lower  award  price 

■  Leads  to  government  repeatedly  changing  scope, 
schedule,  budget  profile,  etc. 

■  Leads  to  five  to  ten  years  later  recognition  “real” 
cost  multiple  of  bid 

■  Leads  to  Nunn-McCurdy  Breach 


Integrity  -  Service  -  Excellence 
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Lessons  Learned 


■  Reward  “Being  Honest” 

■  Budget  to  probable  costs 

■  Break  programs  into  more  manageable  blocks 

■  Recognize  that  TSPR  doesn’t  work 

■  Incorporate  “Back  to  Basics”  strategy 


Integrity  -  Service  -  Excellence 
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“ Back  to  Basics”  Acquisition 


Four-stage  process 

■  Science  &  Technology 

■  Technology  Development 

■  Systems  Development 

■  System  Production 

Apportion  risk  among  the  st 

■  Highest  risk  in  S&T 

■  Base  production  on  mature 
technology  for  lowest  risk 

■  Shorten  cycle  time 
Incremental  deliveries 

■  Block  approach 


STP-R1  Streak 
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Closing 


■  Every  program  is  vulnerable 

■  Keep  it  simple 

■  Close  program  management 

■  Realistic  budgets 


Systems  engineering  underpins  success 


Integrity  -  Service  -  Excellence 
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NDIA 

9th  Annual  Systems  Engineering  Conference 

San  Diego,  California 
October  23-26,  2006 

Long-Term  Product  Support  Requires 
Good  Configuration  Management 


Mark  Pellegrini 
Jeanne  Balsam 
Lee  Sheiner 

Electronic  Systems  Laboratory 
Georgia  Tech  Research  Institute 
Georgia  Institute  of  Technology 
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Georgia  Tech  Research  Institute  (GTRI)  Overview 


•  Unit  of  the  Georgia  Institute  of  Technology 

•  1200+  employees 

•  70%  of  research  employees  hold  advanced  degrees 

•  Wide  variety  of  products 

•  Customers  include  federal  and  state  government; 
and  industry 

•  Competitively  bid  projects  range  greatly  in  size  and 
duration 

•  More  Info:  http://www.qtri.qatech.edu/ 
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Overview 
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Risks 


Some  Configuration 
Manarament  (CM) 
fundamentals  « 

CM  environments 
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What  is  Configuration  Management? 


Controlling  the 
components  of 
a  system 

Controlling  the 
changes  to 
those 

components 


Georgia  1  I 

Tech  UH  Qitd®Sd'S(ui'S® 


LongTermSupportRequiresGoodCM  -  4 


Important  Things  to  Remember  About 
Configuration  Management 

•  Configuration  management  is  not  only  for  us,  but  for  those  who  will 
follow  in  our  footsteps 

•  Software  products  can  consist  of  many  different  types  of 
components 

•  Source  files  used  to  build  the  product 

•  Documentation 

•  Released  executables,  DLLs,  etc. 

•  Third  Party  Components 

•  Bitmaps,  audio  files 

•  Test  data,  test  results 

•  And  so  on.... 

DirasffilH&yru)® 
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Risky  Business 


Some  of  the  Risks 


•  Missing  source  files 

•  Wrong  files  used  in  builds 

•  Missing  build  instructions 

•  Missing  deliverables 

•  Missing  tools  (e.g.  compilers) 

•  Missing  3rd  party  software 

•  Unknown  versions  of  3rd  party  SW  incorporated  into  releases 

•  Undocumented  baselines 

•  Missing  development  environment  information  (e.g.  IDEs) 

Georgia  [^©©©sHrdhi 
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The  Result  of  Realized  Risks 


Releasing  the  wrong  products 

Can’t  create  variants 

Can’t  re-build  releases 

Can’t  continue  development 

Can’t  take  that  vacation  you  were  planning 


Comprehensive  CM  is  Insurance 


Georgia 

Tech  M  OirasffilH&yru)® 


LongTermSupportRequiresGoodCM  -  9 


Development  Environments 


Ideal  Software  Development 
Environment  for  Good  CM 


Budget  surplus 
Very  few  software  releases 
Deliveries  made  ahead  of  schedule 
Many  cross-trained  team  members 
Continuous  development  (no  interruptions) 
Stable  Staff 

Developers  love  to  write  documentation 
Prodigious  amounts  of  CM  documentation 
Software  builders  separate  from  developers 
Relatively  short  duration  with  a  definite  end 


Georgia , 
Tech 
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Worst-Case  Software  Development 
Environment  for  Good  CM 


Tight  budgets 

Frequent  releases  over  many  years 
Just-in-time  deliveries  -  J  j  j 

Compartmented  skills  A  /  ( 

Product  development  frequently  suspende 
High  staff  turnover 

Fesw  developers  (or  a  single  one!)  / 
Nobody  wants  to  write  documentation  / 
Sparse  or  non-existent  CM  documentation 
Developers  build  the  releases 


Georgia 
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Our  Assumed  Environment 


•  Small  project  team 

•  Developers  build  and  maintain  their  own  source  code 
and  executables 

•  Product  is  expected  to  go  through  multiple  releases, 
spanning  years  and  perhaps  decades 

•  Project  team  will  likely  change  over  the  lifecycle  of  the 
product 

•  Product  development  may  become  dormant  during 
lifecycle;  developers  may  move  from  project  to  project 


Long-Term,  Small  Projects  Will 
Stress-Test  Your  CM  Practices 


Georgia 
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Things  Can  Suddenly  Change 


•  Do  not  confuse  the  delivery  of 
correct  products  with  a  process 
that  delivers  the  correct  products 

•  In  other  words,  everyone  is  lucky 
sometimes;  very  few  are  lucky  all 
the  time 

•  In  a  weak  CM  environment,  loss  of 
key  personnel  can  precipitate  a  CM 
disaster 


Georgia 
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“Bus  Number” 


•  The  minimum  number  of  developers  you  would  have 
to  lose  to  seriously  affect  your  project 

•  Higher  is  better 

•  We  prefer  “Lottery  Number” 

•  Don’t  forget  that  buses  can  hit  your  development 
systems,  too 
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Documenting  Baselines 
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Proper  Identification  of  Baselines 


•  All  configuration  items  must  be  recoverable  from 
the  CM  Library 


•  All  configuration  items  must  be  identified 

•  Do  not  include  extraneous  items 

•  Good  planning  (and  re-planning)  helps 
to  make  defining  baselines  simpler 


Georgia 
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Product  Version  Description 


•  Developed  for  all  Product  Baselines 

•  Draft  developed  early  to  capture  critical  information 

*  Documents  not  only  what  is  needed  to  build  the 
product  but  how  to  build  it 

*  Critically  important,  particularly  if  product 
development  is  suspended  and/or  product 
development  team  changes 
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Product  Version  Description: 
Bringing  it  all  together 
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Product  Version  Description: 
Release  &  Build  Descriptions 


Release 

•  Physical  inventory  of 
materials  released 

•  Inventory  of  released 
product  contents 

•  Installation  instructions 

•  Changes  from  previous 
release 

•  Operating  instructions 


Build 

•  Configuration  items  built 

•  Inventory  of  configuration 
items  needed  to  create  the 
built  items 

•  Applications/tools  required 
to  build 

•  Build  procedures 
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Product  Version  Description: 
Other  Information 

•  Instructions  for  building  installation  disk 

•  Include  CM  Library  location  and  version  of  script  files,  if  a  script  is 
used 

•  Known  problems/planned  upgrades 

•  Workarounds 

•  ECRs  (Engineering  Change  Requests) 

In  general,  more  detail  is  better  than  less. 

Someone  who  has  not  developed  the  product  must  be  able  to 
build  it  from  the  PVD! 


1 
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Auditor  Questions 

•  Exactly  what  are  we  delivering? 

•  Which  of  these  delivered  items  are  we 
responsible  for  building? 


•  Where  are  the  components  we  need  to 
build  those  items? 

•  What  are  the  tools  we  need  to  build 
them?  Where  are  these  tools? 


•  How  do  we  build  the  items  we  need  to 
build? 

•  How  do  we  know  we  built  the  right  thing? 
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Independent  Audits  are  Indispensable 
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Verification  of  Baselines 


•  “Clean  Machine”  must  be  used  for  verification  of 
the  integrity  of  the  baseline  as  documented  in  the 
Product  Version  Description 


•  Clean  Machine  must  not  use  networked 
during  build  (unless  specified  by  PVD) 

•  Use  virtual  machines  and  ghost  images 
efficiently  create  Clean  Machines 

•  Development  environment 

•  Runtime  environment 


drives 
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Summary  and  Conclusions 
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Some  Things  to  Remember... 


•  Control  all  items  needed  to  build  the  product  (including 
install  disk),  not  just  source  files 

•  Control  all  tools  needed  to  build  the  product 

•  Control  all  scripts,  IDE  configuration  files,  or  other  files 
used  in  the  build  process 

•  Capture  development  environment  nuances  or  problem 
solutions  in  real-time  -  don’t  wait 

•  Document  baselines  thoroughly,  including  build 
instructions 

•  Audit  baselines  independently,  on  clean  machines;  install 
on  a  clean  machine  too 
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Determine  the  Best  Practices  for  You 


•  Discover  your  weaknesses 

•  Evaluate  what  your  risks  are 

•  Estimate  your  costs  for  failure 

•  Trying  to  address  everything  all  at  once  may  be 
impractical  -  prioritize 

•  Use  approaches  that  are  practical  for  your 
situation 

•  Do  what  makes  sense 
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Wrap-Up 


Contact  Information 


•  Mark  Pellegrini 

•  mark.pelleqrini@gtri.gatech.edu 

•  Jeanne  Balsam 

•  ieanne.balsam@qtri.qatech.edu 

•  Lee  Sheiner 

•  lee.sheiner@gtri.gatech.edu 

•  More  Info  about  GTRI: 

http://www.qtri.qatech.edu/ 
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Top  5  Systems  Engineering  Issues 

within 

DOD  and  Defense  Industry 


Task  Report 


July  26-27,  2006 


NDIA  Systems  Engineering  Division  -  Top  5  SE  Issues 
July  26-27,  2006 


Version  8a  final 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Task  Description 

Identify  Top  5  Systems  Engineering  problems  or  issues 
prevalent  within  the  defense  industry 

•  Update  status  and  re-validate  SE  issues  from  2003 

•  Agree  upon  Top  5  SE  issues  for  2006 

Document  issues 

•  Description  and  current  state 

•  Rationale  and  SE  impacts 

•  Develop  recommendations  (short  term  and  long  term) 

Generate  task  report 

•  Submit  to  OSD  mid-August  2006 
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STRENGTH  THHCHGH  INDUSTRY  &  TECHNOLOGY 


Task  Group  Participants 

18  participants  from  industry  and  government 


Alion  Science  and  Technology 

Boeing  IDS 

Harris 

Lockheed  Martin 
Northrop  Grumman 
Orbital  Sciences  Corp 
Simulation  Strategies 
SPARTA,  Inc. 

Raytheon  SAS 


Marine  Corp  Systems  Command 
National  Defense  University, 
Industrial  College  of  the  Armed 
Forces  (ICAF) 

SAF/AQRE 

Software  Engineering  Institute 
USAF  Center  for  Systems 
Engineering 


•Task  Group  Leads:  Geoff  Draper,  Harris  Corp, 
and  Hal  Wilson,  Northrop  Grumman 
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STRENGTH  THHOIGH  INDUSTRY  &  TECHNOLOGY 


Task  Group  Activities 


Top  5  SE 


Initiatives 


•18  issues 

•Detailed  sub-issues 
•Effects  and  root 
causes 


•Affinity  grouping 
•Nominal  group 
technique 
•Consensus  review 
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STRENGTH  THKOIGH 1NDUSTRV  &  TECHNOLOGY 


Background  -  Top  5  SE  Issues  of  2003 

Top  5  Systems  Engineering  Issues  (2003) 

•  Lack  of  awareness  of  the  importance  and  value,  timing, 
accountability,  and  organizational  structure  of  SE  on 
programs 

•  Adequate,  qualified  resources  are  generally  not  available 
within  Government  and  industry  for  allocation  on  major 
programs 

•  Insufficient  SE  tools  and  environments  to  effectively  execute 
SE  on  programs 

•  Requirements  definition,  development  and  management  is 
not  applied  consistently  and  effectively 

•  Poor  initial  program  formulation 

Details  on  NDIA  Systems  Engineering  Division  web  site 

http://www.ndia.org/Content/ContentGroups/Divisions1/Svstems  Enqineerin 

q/PDFs18/Modelinq  Committee  PDFs/Februarv2003  top  5  issues.pdf 
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STRENGTH  THHOl  GH  INDUSTRY  &  TECHNOLOGY 


Summary  of  OSD  Initiatives  for  SE  Issues 

OSD  Initiative  Summary 

Policy  updates  •  Systems  Engineering  Plan  (SEP)  on  all  programs 

•  OSD  SEP  reviews  for  ACAT  ID  and  1AM  programs 

•  Each  PEO  shall  have  a  lead  or  chief  systems  engineer  who 
monitors  SE  implementation  within  program  portfolio 

•  Event-driven  technical  reviews  with  entry  criteria  and  independent 
subject  matter  expert  participation 

Improved  SE  Guidance  •  Defense  Acquisition  Guidebook 

•  SEP  Preparation  Guide 

•  IMP/IMS  Guide 

•  Risk  Management  Guide 

•  Capability  Maturity  Model  Integration-Acquisition  Module  (CMMI- 
AM) 

Education  and  Training  •  Strengthened  SE  curricula  via  Defense  Acquisition  University 

(DAU) 

•  Certifications 

•  Continuous  Learning  Modules  for  R&M,  Technical  Reviews, 
System  Safety,  M&S  in  SE 

•  Enhanced  career  paths 

System  level  assessments  •  Non-advocate  program  support  reviews 

•  Technical  Planning  Review 
Integrating  Development  T&E  •  Effective  early  engagement 
with  policy  and  assessments 

Outreach  •  SE  Forum 

•  Leverage  partnerships  with  industry  and  academia 

•  Standup  and  participation  in  SE  WIPTs 

Reference:  Systems  Engineering  Update,  NDIA  Top  5  Issues  Workshop.  July  26,  2006.  Briefing  by  Mr.  Robert  Skalamera, 

Deputy  Director.  Systems  and  Software  Engineering  (Enterprise  Development).  Office  of  the  Deputy  Under  Secretary  of  Defense  (A&Ti _ 
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STRENGTH  THROUGH  INDUSTRV  &  TECHNOLOGY 


Assessment  of  Top  5  SE  Issues  from  2003 


Issue  (2003) 

Summary  Assessment  (2006) 

Lack  of  awareness  of  the  importance, 
value,  timing,  accountability,  and 
organizational  structure  of  SE  on 
programs 

•  Awareness  achieved  by  top-down  policy,  guidance,  training, 
organizational  structures,  etc. 

•  Others  aspects,  such  as  buy-in,  recognition  of  value,  or 
behavioral  changes  are  not  yet  consistently  evident. 

•  Measures  of  effectiveness  are  needed 

Adequate,  qualified  SE  resources  are 
generally  not  available  within 

Government  and  industry  for  allocation 
on  major  programs 

•  Qualifications  addressed  via  training  and  certifications,  but  not 
necessarily  the  depth  of  the  pipeline 

•  Availability  of  adequately  skilled  SEs  remains  an  issue  for  both 
government  and  industry  (diminishing  availability  to  support 
growing  needs  and  increasing  complexity) 

Insufficient  SE  tools  and  environments 
exist  to  effectively  execute  SE  on 
programs 

•  Model-based  tools,  methods,  and  repositories  are  improving, 
and  are  applied  in  training 

•  Still  need  better  integration  of  tools  and  methods  across  SE  and 
other  disciplines 

•  Modeling  and  simulation  plans  need  to  be  implemented 

Requirements  definition,  development 
and  management  is  not  applied 
consistently  and  effectively 

•  Improved  guidance  has  yet  to  be  consistently  implemented 

•  Must  address  capabilities  as  well  as  specified  requirements 

Poor  initial  program  formulation 

•  SEP  policy,  guidance,  and  event  criteria  should  improve 
program  plans  and  estimates,  but  effectiveness  not  yet  clear 

•  Greater  coordination  and  integration  of  government/contractor 
plans  needed  early  in  acquisition  life  cycle 

•  Greater  integration  of  interfaces  between  acquisition, 
requirements,  and  programmatics  needed 
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STRENGTH  THKOl  GH 1NDUSTKV  &  TECHNOLOGY 


Top  5  SE  Issues  -  2006 

•Key  systems  engineering  practices  known  to  be  effective  are  not 
consistently  applied  across  all  phases  of  the  program  life  cycle. 

•Insufficient  systems  engineering  is  applied  early  in  the  program  life 
cycle,  compromising  the  foundation  for  initial  requirements  and 
architecture  development. 

•Requirements  are  not  always  well-managed,  including  the  effective 
translation  from  capabilities  statements  into  executable 
requirements  to  achieve  successful  acquisition  programs. 

•The  quantity  and  quality  of  systems  engineering  expertise  is 
insufficient  to  meet  the  demands  of  the  government  and  the  defense 
industry. 

•Collaborative  environments,  including  SE  tools,  are  inadequate  to 
effectively  execute  SE  at  the  joint  capability,  system  of  systems 
(SoS),  and  system  levels. 
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STRENGTH  THHtfl  1NDUSTKV  &  TECHNOLOGY' 


Summary  -  Evolution  of  Top  SE  Issues 


2003  2006 


Lack  of  awareness 
of  SE  importance 

Inconsistent  SE 
practices  across  all 
life  cycle  phases 

Lack  of  adequate, 
qualified  resources 

Insufficient  quantity 
and  quality  of  SE 
expertise 

Insufficient  SE 
tools  and 
environments 

Inadequate  tools 
and  collaborative 
environments 

Inconsistent 

requirements 

definition 

Requirements  not 
well  managed  or 
translated 

Poor  initial 

program 

formulation 

Insufficient  SE 
early  in  the  life 
cycle 

Complex  systems, 
systems  of  systems 


Though  progress  is  being  made,  the  environment  is  changing  and  many  underlying 
issues  still  remain. 
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STRENGTH  THKDI  GH INDUSTRY  &  TECHNOLOGY 


Details  of  Top  5  SE  Issues 

of  2006 
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STRENGTH  THHOIGH  INDUSTKV  &  TECHNOLOGY 


Issue  1:  Key  systems  engineering  practices  known  to  be  effective  are  not 
consistently  applied  across  all  phases  of  the  program  life  cycle. 

Discussion  Points: 

•  Inconsistent  SE  practices  for  program  planning  and  execution 

•  Companies  do  not  always  execute  at  claimed  CMMI®  levels 

•  Confusion  between  architecting,  SE,  frameworks  (DODAF) 

•  Lacking  consistent  SE  measures  (effectiveness,  value,  ROI) 

•  Technical  baselines  not  always  established,  controlled,  and  maintained 

•  Risk  management  inconsistent;  guidance  lacks  specificity,  opportunities 

•  Dependencies  on  external  interfaces  outside  program  control  (e.g.,  SoS) 

Recommendations: 

Ensure  institutionalization  of  effective  SE  practices  into  program 
planning  and  execution 

•  Define  and  monitor  required  SE  practices  (e.g.,  DAG)  for  each  phase 

•  Strengthen  CMMI®  for  deployment  of  appraised  processes  on  all  programs 

•  Require  appropriate  SE  measures  across  the  life  cycle  (e.g.,  PSM,  LAI) 

•  Require  use  of  DOD  Risk  Management  Guide,  per  joint  working  group 

•  Define  and  promulgate  interface  change  control  processes  for  SoS 

(DCMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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STRENGTH  THHOKjH  INDUSTRV  &  TECHNOLOGY 


Issue  2:  Insufficient  SE  is  applied  early  in  the  life  cycle,  compromising 
the  foundation  for  initial  requirements  and  architecture  development. 

Discussion  Points: 

•  Requirements,  budgets,  schedules  established  without  adequate  SE. 

•  Optimistic  cost  and  schedule  estimating. 

•  SE  and  PM  functions  are  not  well  integrated,  roles/responsibilities  unclear. 

•  SE  compromised  in  planning  and  design  when  SE  value  not  understood. 

•  SoS  inter-program  complexity  not  accommodated  in  SE  planning. 

Recommendations: 

Integrate  engineering  planning  within  the  acquisition  life  cycle  to  ensure 
adequate  time  and  effort  for  SE  early  in  the  program  life  cycle. 

•  Expand  SEP  policy  to  address  risks  of  inadequate  SE  prior  to  Milestone  A 

•  Establish  guidance  on  SE/PM  roles,  responsibilities,  communications 

•  PMs  ensure  distribution  of  SE  effort  among  prime  and  subcontractors 

•  Increase  PM  awareness  of  value  of  effective  SE 

•  Provide  guidance  for  SE  activities  needed  for  SoS 

•  Integrate  acquisition  processes  with  requirements,  programmatics  to 
ensure  procurement  is  consistent  with  schedule,  resources,  constraints 

•  Expand  independent,  non-advocate  reviews  of  program  formulation 
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STRENGTH  THHOIGH  INDUSTKV  &  TECHNOLOGY 


Issue  3:  Requirements  are  not  always  well-managed,  including  the 
effective  translation  from  capabilities  statements  into  executable 
requirements  to  achieve  successful  acquisition  programs. 

Discussion  Points: 

•  Management  structures  and  roles  not  well  understood  for  SoS,  capabilities 

•  Critical  common  element  areas  across  capability  areas  must  be  identified,  managed 

•  Capabilities  broaden  engineering  trade  space  due  to  less  specific  requirements. 

•  Do  not  always  have  adequate  access  to  trained  SE  resources  for  defining  capabilities 

•  SoS  environment  strains  capability  process  due  to  extended  complexity,  governance 

•  Inadequate  understanding  of  true  capabilities  needed  by  warfighters 

•  Translating  SoS  capabilities  into  insertable  technology  across  legacy  systems  requires 
series  of  feedback  loops 

Recommendations: 

Emphasize  the  application  of  SE  practices  and  resources  to  the  capability  definition 
process  to  address  warfighter  needs  and  translation  into  executable  programs 

•  Lead  the  effort  to  require  roles  and  structure  that  include  SE  in  capabilities  definition 

•  Standardize  terms,  processes,  and  methods  for  SoS  and  enterprise  SE 

•  Enhance  the  application  of  SE  to  capabilities  development  to  better  define  systems 
elements  common  across  capabilities 

•  Emphasize  technical  planning  for  systems  adaptability  and  technology  insertion  from 
program  onset 

•  Emphasize  the  need  for  expanded  contractor  communications  and  interactions  in  early 
requirements  definition  process 

•  Encourage  adoption  of  CMMI  best  practices  for  requirements  (acquisition,  development) 
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STRENGTH  THKOL  GH 1NDU1STRV  &  TECHNOLOGY 


Issue  4:  The  quantity  and  quality  of  systems  engineering  expertise  is 
insufficient  to  meet  the  demands  of  the  government  and  the  defense 
industry. 

Discussion  Points: 

•  Short  supply  of  experienced,  trained  SE  in  government  and  industry. 

•  PM/SE  turnover  and  fewer  opportunities  for  OJT. 

•  Inconsistent  SE  certification  methods  across  government  and  industry. 

•  Inconsistent  roles  and  responsibilities  across  services  results  in  inefficient 
use  of  government  and  industry  resources 

Recommendations: 

Grow  SE  expertise  through  training,  career  incentives,  and  broadening 
“systems  thinking”  into  other  disciplines. 

•  Modify  SEP  template  to  clearly  specify  required  SE  qualifications  for 
program  staffing 

•  Work  with  academia  to  include  introductory  SE  course  in  degree 
programs,  establish  recommended  curricula  for  consistent  skills 

•  Establish  SE  job  code  and  career  incentives  for  promotion  into  PM 

•  Require  “systems  thinking”  training  for  other  disciplines  and  PM  to  expand 
SE  candidate  pool 

•  Encourage  an  integrated  and  consistent  SE  certification  definition  between 
industry  and  govt. 
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Issue  5:  Collaborative  environments,  including  SE  tools,  are  inadequate 
to  effectively  execute  SE  at  the  joint  capability,  system  of  systems  (SoS), 
and  system  levels. 

Discussion  Points: 

•  Collaborative,  information-sharing  environments  are  needed  to  enable  all 
stakeholders  to  work  together  effectively  (M&S,  workflow,  design  tools) 

•  Policies,  guidance,  standards  inadequate  for  interoperability/sharing/reuse 

•  Expanded  complexity  of  the  SE  trade  space  due  to  capabilities,  SoS,  PBA, 
etc.  exceeds  capabilities  of  current  tools  and  environments  (M&S,  MBSE) 

Recommendations: 

Strengthen  and  clarify  policy  and  guidance  regarding  use  of  collaborative 
environments,  models,  simulations,  and  other  automated  tools. 

•  De-conflict  policies  and  standards  to  enable  data  sharing,  tool  interoperability 

•  Establish  partnerships  and  roadmaps  for  evolving  tool  capabilities 

•  Define  best  practices  and  education  to  foster  use/sharing  of  tools  and  data 

•  Sponsor  definition  of  and  encourage  the  use  of  a  DoD  business  model 
facilitating  shared  information  and  tools 

•  Implement  Acquisition  Modeling  and  Simulation  Master  Plan 
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Task  Description 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Identify  Top  5  Software  Engineering  problems  or  issues 
prevalent  within  the  defense  industry 

•  NDIA  Workshop  August  24-25,  2006,  Washington  D.C. 

Document  issues 

•  Description  and  current  state 

•  Rationale  and  SW  impacts 

•  Develop  recommendations  (short  term  and  long  term) 

Generate  task  report 

•  Submit  to  OSD 
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Task  Group  Participants 
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26  workshop  participants  from  industry,  government,  academia 

•  Additional  inputs  received  via  email 


BAE  Systems 

The  Boeing  Company 

Center  for  Strategic  and  Inti.  Studies 

Computer  Sciences  Corporation 

Department  of  Defense  (DoD) 

Defense  Contract  Mgmt.  Agency  (DCMA) 
Florida  Institute  of  Technology 
Harris  Corporation 
Jacobs  Technology 
Lockheed  Martin  Corporation 


Massachusetts  Institute  of  Technology  (MIT) 
The  MITRE  Corporation 
Naval  Air  Systems  Command 
OUSD  (A&T) 

Open  Systems  Joint  Task  Force 
Raytheon  Company 
RDECOM  ARDEC 

Asst.  Secy,  of  the  Air  Force  (Acq),  SE  Division 
Software  Engineering  Institute  (SEI) 

Systems  and  Software  Consortium 

U.S.  Air  Force  Aeronautical  Systems  Center 


Task  Group  Leads: 

Geoff  Draper  (Harris) 

Jeff  Dutton  (Jacobs  Technology) 
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Task  Group  Activities 


Top  5  SE 
Issues 

(2006) 

- ► 


Status  of 
OSD 
Initiatives 
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r 


Identify 
Candidate 
SW  Issues 


Candidate 

issues 


Prioritize 
SW  Issues 
and  Actions 


Top  5  SW 
Issues 


Document 
Top  5  SW 
Issues  for 
OSD  Report 


Task 

report 


•Background 
•Brainstorming 
•Group  discussion 
•85  root  issues  identified 


•30  consolidated  root  issues 
•Grouped  and  refined 
•Effects  and  root  causes 
•Consensus  review,  voting 


•Extended  to  7  key  issues 
•Detailed  description 
•Recommendations 
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Common  Themes  Noted 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Increasing  demands  for  rapid  response  to  changing  threats 

•  New  warfighter  capabilities 

•  Unprecedented  complexity 

•  Increasing  dependence  on  large,  software-intensive  systems 

Evolving  acquisition  environment 

•  Innovative  and  aggressive  acquisition  strategies 

•  Extensive  reuse  of  legacy  components  and  system  portfolios 

•  Reductions  in  acquisition  work  force  due  to  force-shaping 
measures 

Current  approaches  for  acquiring,  developing,  verifying, 
and  sustaining  software  enabled  systems  are  inadequate 
to  deal  with  the  complexities  of  a  dynamic  and  changing 
acquisition  environment. 
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Top  Software  Issues 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  The  impact  of  requirements  upon  software  is  not  consistently 
quantified  and  managed  in  development  or  sustainment. 

2.  Fundamental  system  engineering  decisions  are  made  without  full 
participation  of  software  engineering. 

3.  Software  life-cycle  planning  and  management  by  acquirers  and 
suppliers  is  ineffective. 

4.  The  quantity  and  quality  of  domain-knowledgeable  software 
engineering  expertise  is  insufficient  to  meet  the  demands  of 
government  and  the  defense  industry. 

5.  Traditional  software  verification  techniques  are  costly  and 
ineffective  for  dealing  with  the  scale  and  complexity  of  modern 
systems. 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure 
execution  of  complex  software  in  distributed  environments. 

7.  Inadequate  attention  is  given  to  total  lifecycle  issues  for 
COTS/NDI  impacts  on  lifecycle  cost  and  risk. 
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Issue  1:  The  impact  of  system  requirements  upon 
software  is  not  consistently  quantified  and  managed 
in  development  or  sustainment. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Discussion  Points: 

•  Impacts  of  changes  to  requirements  not  addressed  consistently. 

•  SW  requirements  are  not  always  adequately  addressed  in  system  level 
requirements  or  solicitations/contracts. 

•  Conditions  leading  to  SW  reqts  changes  are  not  clearly  understood. 

•  Weak  linkage  to  SW  requirements  for  capabilities,  portfolios. 


Recommendations: 

Enforce  effective  software  requirements  development  and 
management  practices,  including  assessment  of  change  impacts, 
for  both  the  acquirer  and  supplier  organizations. 

•  Strengthen  policies/guidance  for  full  requirements  traceability. 

•  Evaluate  effectiveness  of  models  for  estimating  SW  and  changes. 

•  Improve  involvement  of  SW  and  acquisition  expertise 

-  Early  requirements  definition  (JCIDS) 

-  Balance  user  requirements  with  cost/schedule  constraints 

-  Impact  assessment  and  communication  of  requirement  changes 
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Issue  2:  Fundamental  system  engineering  decisions 
are  made  without  full  participation  of  software 
engineering. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Discussion  Points: 

•  SW  not  consistently  involved  in  architectural  decisions  early  in  the  life  cycle. 

•  Must  have  authority  to  participate  in  system-level  decision  making  and  trades. 

•  SE  and  SW  life  cycles,  processes,  and  products  are  not  always  consistent  or 
harmonized  for  meaningful  cross-discipline  integration  to  occur. 

•  Segregation  of  SE/SW  proposal  inputs  can  impede  coordination. 

•  System  development  methods  do  not  properly  leverage  SW  ability  to  rapidly 
field  enhanced  capabilities. 


Recommendations: 

Institutionalize  the  integration  and  participation  of  software 
engineering  in  all  system  engineering  activities. 

•  Designate  an  empowered  SW  architect  (acquirer  and  supplier)  early. 

•  Update  policies,  standards,  guidance  to  emphasize  SW  in  decision-making. 

•  Promote  effective  SE/SW  communications  and  cross-training. 

•  Adopt  acquisition  strategies  and  development  approaches  that  leverage  rapid 
SW  development  cycles  (iterative  capability,  sustainability). 
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Issue  3:  Software  life  cycle  planning  and 
management  by  acquirers  and  suppliers  is 
ineffective. 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Discussion  Points: 

•  SW  risks  and  life  cycle  costs  are  not  consistently  accommodated  in  planning. 

•  Realistic  schedule  and  effort  (cost)  estimates  are  often  rejected  or  constrained. 

•  Pressure  to  rapidly  procure  new  capabilities  can  inhibit  balance  of  life  cycle 
cost,  schedule,  performance  expectations  to  achieve  executable  programs. 

•  Lack  of  SW  expertise  applied  in  planning  and  management. 

•  SW  processes/methods  improperly  selected,  tailored,  executed,  monitored. 

•  SW  measures  are  not  used  effectively  or  acted  upon. 

Recommendations: 

Establish  a  culture  of  quantitative  planning  and  management,  using 
proven  processes  with  collaborative  decision-making  across  the 
software  life  cycle. 

•  Establish  collaborative  contractual  relationships  enabling  joint  ownership  of 
issues. 

•  Change  culture  to  base  expectations  on  historical  data  and  realistic  constraints. 

•  Emphasize  selection  of  appropriate  processes/methods  and  codify  in  contract. 

•  Incentivize  life-cycle  perspective  and  iteration  to  circumvent  near-term  thinking. 
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Issue  4:  The  quantity  and  quality  of  domain-knowledgeable 
software  engineering  expertise  is  insufficient  to  meet  the 
demands  of  the  government  and  the  defense  industry. 

Discussion  Points: 

•  Insufficient  highly  skilled  domain-knowledgeable  SW  professionals  to  meet  the 
growing  demand,  due  to  inadequate  funding,  insufficient  career  incentives, 
competition,  downsizing,  etc. 

•  Insufficient  infrastructure  to  rapidly  transition  skilled  staff  to  new  programs. 

•  Inadequate  SW  staffing  across  the  system  life  cycle  (architecture,  trades,  etc.) 

•  Career  path  incentives  are  insufficient  to  attract  SW  professionals. 

Recommendations: 

Collaborate  on  innovative  strategies  to  staff  to  appropriate  levels 
and  to  attract,  develop,  and  retain  qualified  talent  to  meet  current 
and  future  software  engineering  needs  in  government  and  industry. 

•  Develop  a  Software  Competency  Roadmap,  with  strategic  decisions  about  the 
future  workforce  (competencies,  downsizing,  R&D,  etc.) 

•  Establish  joint  government,  industry,  academia  working  group  to  identify 
strategies  for  expanding  SW  resource  pool  and  skill  sets. 

•  Cross-train  among  functional  disciplines  (PM,  SE,  etc.)  to  expand  SW 
engineering  pool  and  extent  of  domain  knowledge. 
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Issue  5:  Traditional  software  verification  techniques 
are  costly  and  ineffective  for  dealing  with  the  scale 
and  complexity  of  modern  systems. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Discussion  Points: 

•  Over-reliance  on  testing  alone  rather  than  robust  SW  verification  techniques. 

•  Manual  testing  techniques  are  labor-intensive,  scale  poorly,  and  are 
unproductive  relative  to  the  large  investment  of  resources. 

•  Compliance-based  tests  do  not  adequately  cover  risks  or  failure  conditions. 

•  Tests  are  over-documented  with  disproportionate  effort  on  detailed  procedures. 

•  Education,  training,  certifications  are  inadequate  to  develop  effective  test  skills. 

Recommendations: 

Study  current  software  verification  practices  in  industry,  and 
develop  guidance  and  training  to  improve  effectiveness  in  assuring 
product  quality  across  the  life  cycle. 

•  Sponsor  a  study  of  state-of-the-practice  verification  and  testing  approaches. 

•  Review/update  testing  policies  and  guidance  to  emphasize  robust,  productive 
approaches  that  maximize  ROI. 

•  Review  adequacy  of  verification  plans/approaches  early  in  the  acq.  life  cycle. 

•  Emphasize  skilled  investigation  throughout  the  life  cycle,  based  on  coverage, 
risk  mitigation,  high  volume  automation. 

•  Strengthen  curricula,  training,  certifications,  career  incentives  for  testing  roles. 
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Issue  6:  There  is  a  failure  to  assure  correct, 
predictable,  safe,  secure  execution  of  complex 
software  in  distributed  environments. 


i  >  r.A  > « ;  r.i  »t  i  ^  ^  i  *1'^  > f.i  <r.»yiaiiuLgM 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Discussion  Points: 

•  SW  is  inherently  vulnerable  to  widespread  SW  assurance  threats  -  we  must  be 
confident  in  the  supply  chain  pedigree. 

•  Current  techniques  are  inadequate  to  verify  assured  components  with  well- 
understood  properties. 

•  Assurance  of  systems,  and  SoS,  cannot  be  easily  inferred  from  components 
due  to  issues  such  as  composability,  emergent  behavior. 

•  Exhaustive  testing  to  rule  out  vulnerabilities  is  not  feasible. 

Recommendations: 

Collaborate  with  industry  to  develop  approaches,  standards,  and 
tools  addressing  system  assurance  issues  throughout  the 
acquisition  life  cycle  and  supply  chain. 

•  Establish  collaborative  initiatives  to  develop  and  deploy  comprehensive  system 
assurance  approaches. 

-  Resistance  to  intrusion  and  compromise 

-  Commercial  standards  to  address  vulnerability  throughout  the  supply  chain 

•  Define  SW  assurance  quality  attributes  for  addressing  in  architectural  trades. 

•  Sponsor  system  assurance  research,  policy,  guidance,  training.. 
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Issue  7:  Inadequate  attention  is  given  to  total  lifecycle 
issues  for  COTS/NDI  impacts  on  lifecycle  cost  and 
risk. 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Discussion  Points: 

•  Inadequate  estimating  methods  for  COTS/NDI  (reuse,  open  source,  GOTS,  ...) 

•  COTS/NDI  best  practices  are  known  but  not  consistently  implemented. 

•  Inadequate  attention  to  sustainment  issues  early  in  the  life  cycle. 

•  Open  source  licensing  issues  can  expose  organizations  to  liability,  loss  of  data 
rights,  potentially  large  rework. 

•  Customer  expectations  for  customization  reduce  benefits  of  COTS  solutions. 

•  Insufficient  infrastructure  to  create  and  facilitate  reuse  across  organizations. 

Recommendations: 

Improve  and  expand  guidelines  for  addressing  total  lifecycle 
COTS/NDI  issues. 

•  Encourage  adoption  of  COTS/NDI  best  practices  (SEI,  AIAA,  etc.) 

•  Ensure  COTS/NDI  life  cycle  processes  are  addressed  in  program  plans. 

•  Review  COTS/NDI  life  cycle  support  issues  and  tradeoffs  at  program 
milestones  and  reviews. 

•  Ensure  COTS/NDI  issues  are  addressed  in  OSD  policy  and  SW  Assurance 
initiative. 
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Summary  Recommendations 

Top  SW  Issues  2006 
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STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


1.  Enforce  effective  software  requirements  development  and 
management  practices,  including  assessment  of  change  impacts,  for 
both  the  acquirer  and  the  supplier  organizations. 

2.  Institutionalize  the  integration  and  participation  of  software 
engineering  in  all  system  engineering  activities. 

3.  Establish  a  culture  of  quantitative  planning  and  management,  using 
proven  processes  with  collaborative  decision-making  across  the 
software  life  cycle. 

4.  Collaborate  on  innovative  strategies  to  staff  to  appropriate  levels,  and 
to  attract,  develop,  and  retain  qualified  talent  to  meet  current  and 
future  software  engineering  needs  in  government  and  industry. 

5.  Study  current  software  verification  practices  in  industry,  and  develop 
guidance  and  training  to  improve  effectiveness  in  assuring  product 
quality  across  the  life  cycle. 

6.  Collaborate  with  industry  to  develop  approaches,  standards,  and  tools 
addressing  system  assurance  issues  throughout  the  acquisition  life 
cycle  and  supply  chain. 

7.  Improve  and  expand  guidelines  for  addressing  total  lifecycle 
COTS/NDI  issues. 
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DoD  Weapons/Explosive  Safety  Service  -  Unique,  Inconsistent,  Duplicate 
and  Common  Testing  Standards/Requirements  -  Are  Yours  Here? 

An  OSD  Study  Led  by  the  U.S.  Marine  Corps  Systems  Command 


October  25,  2006 


Presentation  to: 

9th  Annual  Systems  Engineering  Conference 

San  Diego,  CA 


Presentation  by: 

Scott  Rideout,  Safety  Officer 
U.S.  Marine  Corps  Systems  Command 
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Problem 


►  The  Services’  safety  review  boards  use  different  safety  testing  criteria. 

►  When  one  Service  wants  to  use  another  Service’s  existing  weapon  system,  additional  safety 
testing  is  typically  required  by  the  new  Service’s  safety  board  when  the  equipment  has  already 
been  approved  by  a  safety  board.  Some  of  these  tests  are  essentially  redundant. 

►  Common  safety  standards  become  of  greater  importance  as  the  Services  develop  more 
explosive  weapon  systems  with  planned  use  in  a  joint  operating  environment. 

►  Study  was  initiated  with  the  following  long  term  goals 

-  Determine  a  common  definition  of  "acceptable"  weapon  safety  test  criteria  and  acceptable 
results  for  each  and  every  possible  environment  the  weapon  could  be  certified  for. 

-  Develop  a  common  understanding  of  the  attributes  that  constitute  a  safe  weapon. 
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Phase  I  Objectives 


►  Clearly  define  the  requirements  for  explosive  weapon  systems  safety  tests  and  execution 
methodologies  required  by  the  Services  and  Service  review  boards. 

-  Identify  safety  tests  across  the  Services 

-  Analyze  safety  tests  among  different  operating  environments  and  within  each  Service 

►  Assess  requirements  for  current  safety  tests  to  identify  common,  inconsistent,  duplicate  and 
singular  tests. 

-  What  are  the  drivers  for  inconsistent  and  duplicate  safety  tests 

-  What  are  the  processes  behind  inconsistent  and  duplicative  safety  test 

-  Identify  Service  specific  board  processes 

►  Identify  specific  differences  in  safety  tests  between  the  Services. 

-  Identify  differences  in  how  the  tests  are  conducted  and  measured  by  each  Service 

-  Identify  the  drivers  for  the  differences  resulting  in  opportunities  and  benefits 

►  Identify  potential  opportunities  or  benefits. 

-  What  are  the  facing  opportunities  for  fixing  duplicate  and  inconsistent  tests? 

►  Gather  content  on  the  known  and  existing  problems  that  are  faced  in  the  field,  potentially  resolved 
by  common  safety  tests. 
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What  I  Need  from  You 


►  Needs 

-  Do  you  agree  with  the  approach? 

-  Do  the  modes  cover  all  operational  environments? 

-  Where  do  you  see  /  have  you  seen  differences  in  safety  testing  requirements  among  the 
Services? 

-  Where  have  you  seen  differences  in  implementation  of  the  same  safety  testing  requirements? 

►  Contacts 

-  Paige  Ripani,  Booz  Allen  Hamilton,  (703)  412-7702,  ripani_paige@bah.com 

-  Kristin  Norris,  Booz  Allen  Hamilton,  (540)  288-5078,  norris_kristin@bah.com 
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Hypothesis  Statement 


►  Given  the  increasingly  joint  nature  of  American  military  deployment,  services  are 
increasingly  hindered  and  delayed  by  the  current  need  to  require  duplicate  and 
inconsistent  safety  tests  in  order  to  qualify  for  use  and  shipment  to  a  deployed  site. 


Navy/U 
Unique 


Amy- 


PERCEIVED 

END-STATE 


Identified  service- 
specific  safety 
tests 


■orce 
Jnique 


I 
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Work  Plan  and  Detailed  Approach 


►  Data  gathering  of  existing 
and  required  DOD  weapons 
safety  tests. 

►  Interviews  with  safety  board 
representatives,  program 
managers,  system  safety 
leads  and  service-specific 
testers  to  confirm  tests 
identified,  gather  additional 
tests  and  identify 
differences  among  the 
services  about  how  the 
tests  are  setup, 
implemented  and  assessed. 


►  Data  analysis  for  potential  ►  Vetting  of  conclusions  based 

test  duplication,  on  the  analysis  and 

inconsistency  and  single  collection  efforts. 

service  instances  exist. 

►  Select  comprehensive 
sample  of  data  collected  by 
the  team. 


No  Stovepipe  Mentality  -  Approach  is  Representative  of 
All  Types  of  Systems  for  All  Services. 


►  Relational  database  to 
assist  with  the  gathering 
and  organization  of  data. 
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Data  Structure  Hierarchy  provides  the  means  to  compare  the  Mode  to 
the  Test  Classification 


Service 

ARMY,  NAVY,  USMC,  AIR  FORCE 


Database  Structure 


AUR/Component 

(e.g.,  Fuze,  Submunitions,  Explosive) 


Mode 

(e.g.,  Transportation,  Storage/Stowage,  Handling) 


Test  Classification 

(e.g.,  EEE,  Shock,  Vibration,  Thermal) 


Test  Document 

(e.g.,  MIL-STDs,  STANAGs,  AECTPs,  ITOPs) 


Test  Name 

(e.g.,  Jolt,  Temperature  and  Humidity,  Transportation  Vibration,  Salt  Fog) 


Test  Parameters 

(e.g.,  Method,  Temperature,  Pass/Fail) 


Database 

Relationships 


Relationship  of 
Mode  to  Test 
Classification 
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The  database  provides  insight  into  identifiable  inefficiencies  (details  in  the 
following  pages)  by  service... 


►  Current  data  is  in  the  form  of  picture  and  text  based  documents  that  was  parsed  to  fit  into  the  database  tables 

►  Utilizing  a  complex  set  of  queries  the  database  was  produced  a  detailed  cross-sectional  report  of  the  requirements  for  common, 
inconsistent,  duplicate  and  singular  tests  used  within  the  services 

►  The  database  output  can  be  easily  incorporated  into  any  MS  Office  application  for  refined  graphical  analysis  or  added  report 
building  capabilities 


Handling 


Transportation 
Stowage/Storage 


Operational  Use 


SERVICE 


NAVY  USMC  ARMY 


[  Shock 

Mechanical 

1  —  — 

—  — 

—  —  - 

■■  ■■  ^ 

Acoustic 

Thermal 

0 

0 

0  j 

|  Etc 

_ i 

Analysis  According  to  Test  Classification 


►  The  database  houses  information  according  to  Mode  and  Test 
Classification 

►  The  data  detailed  by  Test  Classification  allows  for  comparison  by 
all  similar  test  types  (shock  for  example),  within  and  across  all 
military  services 

►  The  replication  of  similar  tests  by  Test  Classification  expands  when 
tests  within  a  certain  classification  (Mechanical)  also  include  test 
types  within  another  Test  Classification  (Acoustic) 


Bqqz  I  Allen  I  Hamilton 


10 


...and  identifiable  inefficiencies  by  Mode 


Handling 
Transportation 
Stowage/Storage 


Operational  Use 
SERVICE 

Q Shock 
Mechanical 
Acoustic 
Thermal 
HEtc 


NAVY  USMC  ARMY 


AIR 

FORCE 


0 

i 

1 

0 

1 

0 

1 

1 

1 

■ 

1 

1 

- 1 

1 

l 

Analysis  According  to  Mode 


►  The  highest  level  of  duplication  identified  to  date  is  where  the 
Mode  and  the  Test  Classification  are  the  same 

►  Modes  are  expected  to  be  repeated  across  the  military  services, 
however  multiple  tests  within  a  service  of  Mode  and  test 
classification  should  be  considered  Duplicate 

►  Comparisons  are  made  by  all  armament  type,  All,  Fuze, 
Submunition,  Explosive,  Rocket  Motor,  etc. 
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Data  Analysis  Methodology 
hypothesis 


Form  Hypothesis  based  on  Database  Content 


N  ~Z 


■F  T  ransportati 
^  Operational  Use 

W 

X 

X  N 

X 

X 

/ 

/  N 

X  N 

AUR  / 

Component 

Level 

Fuze 

z. 

CO 

Submunition 

1 

— 

^2 

x,  Y,  Z 

Explosive 

Rocket  Motor 

( 

y) 

Y 

AUR 

X,  Y 

X,  Y 

Y 

Safety 

NAVY  USMC 

ARMY 

AIR 

Assumptions 


►  The  Army  has  the  highest  level  of  duplication  evident  in  the 
database 

►  Duplication  and  Redundancy  (if  any)  may  be  the  result  of 
identified  needs  over  time 


Database  supports  validation  of 


Set-Approach  to  Find  Proof  Hypothesis  is  Correct 


Army 

Navy 

USMC 

Air 

Force 

ALL 

Mode 

Handling 

0 

0 

0 

0 

Operational  Use 

0 

0 

0 

Packaging 

Storage/Stowage 

Transportation 

0 

0 

ALL  UP  ROUNDS/COMPONENTS 
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Scoring  Methodology:  Five  different  weights  were  employed  to 
gain  a  high  level  sense  of  the  varying  levels  of  duplication... 


Where  the  Database  Identifies  Duplication,  the  Level 


v  / 


of  Duplication  is  Addressed  using  the  Scoring  Criteria 


Scoring  Criteria 

4 

Duplication  of  effort  identified 

3 

Potential  duplication 

2 

Limited  duplication 

1 

Marginal  duplication 

0 

No  duplication  evident 

...Yielding  initial  proof  of  the  hypothesis  that  duplication  exists 


Assumptions 


►  The  Army  has  the  highest  level  of  duplication  evident  in  the  database 

►  Duplication  and  Redundancy  (if  any)  may  be  the  result  of  identified  needs  over  time 
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Definitions  of  Key  Terms 


►  Opportunity  -  Where  duplicate  and  inconsistent  test  requirements  exists,  causing  inefficiencies, 

loss  or  waste  and  providing  an  opportunity  for  improvement. 

►  Benefit  -  Where  common  and  singular  test  requirements  exists. 

►  Test  Types 

-  Common  -  More  than  one  Military  Service  uses  the  same  safety  test,  test  parameters  and 
test  parameter  values  given  the  same  AUR/Component,  mode  and  test  classification. 

-  Inconsistent  -  More  than  one  Military  Service  uses  the  same  safety  test  and  test  parameters 
and  at  least  one  of  the  test  parameter  values  is  different  given  the  same  AUR/Component, 
mode  and  test  classification. 

-  Duplicate  -  More  than  one  Military  Service  uses  different  safety  tests  for  the  same 
AUR/Component,  mode  and  test  classification.  Different  safety  tests  may  be  driven  by  the 
following  reasons: 

■  Lack  of  coordination,  knowledge  or  focus  on  joint  requirements  during  development 

■  Higher  levels  of  rigor  applied  to  one  test  over  another 

■  Programmatic  legacy 

■  Unique  mission  environment 

-  Singular  -  Only  one  Military  Service  uses  the  safety  test  for  the  same  AUR/Component  and 
test  classification  and  either  the  same  or  different  mode. 
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AUR/Components 


►  System,  Subsystem,  All  (Generic) 

►  System-Specific 

►  Ammunition 

►  Cannon 

►  Electric  Initiators 

►  Explosives 

►  Fuze 

►  Power  Sources 

►  Rocket  Motors 

►  Software 

►  Submunitions 

►  Unmanned  Targets 


-  Field  Mortar  Munitions 

-  Munitions  Carried  in  Tracked  Vehicles 

-  Munitions  with  EEDs 

-  Shipboard  Equipment 

-  Shipboard  Machinery  Equipment 

-  Surface  Launched  Munitions 

-  Underwater  Launched  Munitions 

-  Underwater  Munitions 


AUR/Components  were  determined  by  the  safety  testing  documents. 
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Modes 


►  Transportation 

►  Handling 
-  Vertrep  and  Conrep 

►  Packaging 

►  Storage/Stowage 

►  Developmental 

►  Operational  Use 


-  Road  (Tracked/Wheeled  Land  Vehicles) 

-Rail 

-  Air  (Fixed/Rotary  Wing  Aircraft) 

-  Sea 

-  Grey  Bottom  Surface  Ships  (Operational  Navy  Vessels) 

-  Black  Bottom  (P repo/Merchant  Marine/Commercial) 

-  Undersea 

-  Man  Carried 


-  Open  (e.g.,  shipboard  topside,  pier,  forward  deployment) 

-  Protected  and/or  Environmentally  Controlled  (e.g., 
shipboard  magazine,  ground  magazines) 


-  Man  Carried 

-Tracked/Wheeled  Land  Vehicles 

-  Fixed/Rotary  Wing  Aircraft 

-  Submarine/Undersea 


Ground  Rules 

(1)  Mode  is  determined  by  when  the  mode  will  actually  influence  the  item;  not  when  the  item  will  be  affected  by 
the  influence  of  the  mode. 

(2)  Tests  assigned  to  the  Developmental  Mode  define  the  characteristics  of  the  item;  are  not  typically  tested  in  a 
shipping  or  operational  configuration;  and  do  not  simulate  a  mechanical,  climatic  or  electrical  environment. 
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Safety  Test  Classifications 


►  Chemical  Compatibility 

►  Contamination 

►  Corrosion 

►  EEE 

►  Electrical 

►  ESD 

►  Explosive  Characteristics 

►  Function 

►  Icing 

►  Impact 

►  Initiation 

►  Insensitive  Munitions 

►  Leak  (external) 

►  Leak  (internal) 

►  Lifting 


►  Pressure  -  High 

►  Pressure  -  Low 

►  Safe  Separation 

►  Sequential 

►  Shock 

►  Shock  -  Acoustic 

►  Shock  -  Mechanical 

►  Shock  -  Thermal 

►  Shock  and  Temperature 

►  Shock/Vibration 

►  Shock-Mechanical  (long  drop) 

►  Shock-Mechanical  (short  drop) 

►  Software 

►  Storage  -  Long  Term 

►  Temperature 


►  Temperature  -  Extreme 

►  Temperature  -  High 

►  Temperature  -  Low 

►  Temperature  and  Humidity 

►  Temperature  Shock  Humidity 

►  Transportation 

►  Unknown 

►  Various 

►  Vibration 

►  Wear/Fatigue 


Ground  Rule  -  Tests  assigned  a  test  classification  must  simulate  a  mechanical,  climatic  or 

electrical  environment. 
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Agenda 


►  Study  Overview 

►  Methodology 

►  Data  Collection  and  Analysis 

-  Data  Collection 

-  Data  Analysis 

►  Results  and  Conclusions 

►  Next  Steps 
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Data  Collection  and  Analysis  -  Data  Collection 


►  Coordinated  with  Stakeholders 

-  Briefed  JWSTAP  -  June  06 

-  Briefed  DDESB  Seminar  -  August  06 

-  Briefed  NDIA’s  Systems  Engineering  Conference  -  October  2006 

►  Selected  Comments  from  Selected  Service  SMEs,  Safety  Board  Members  and  POCs  at 
Testing  Facilities 

-  Preston  Parker,  AAC/SES  Eglin  AFB 

■  The  Air  Force  will  not  accept  any  other  Service’s  Safety  Certification  “Carte  Blanche”. 

-  Sharon  Craven,  Navy  Explosive  Qualifications 

■  The  Navy  will  consider  archived  explosives  that  have  been  proven  safe  for  use  over  many  years; 
however,  instead  of  qualifying  it  -  the  explosives  would  be  Final  (Type)  Qualified  only  for  use  under 
certain/individual  circumstances.  The  explosives  would  not  be  considered  “Qualified.” 

-  Dr.  Rao  Surapaneni,  Co-Chair  Army’s  Energetic  Materials  Qualification  Board 

■  MIL-STD-1751A  is  still  being  used  along  with  AOP-7  primarily  because  it  is  uncertain  if  AOP-7 
captures  all  of  the  MIL-STD  requirements. 

-  Chau  Nguyen,  Army’s  Explosives  Safety  and  Quality  Assurance  Department 

■  The  Navy  will  not  deviate  from  specific  Gap  Card  test  requirements  that  they  have  done  forever. 

-  Jeff  Craven,  Testing,  Army’s  Huntsville,  AL 

■  There  are  subtle  differences  in  the  implementation  of  EEE  tests  due  to  available  testing  equipment, 
facility  limitations,  or  that  it  has  always  been  done  that  way. 
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Data  Collection  and  Analysis  -  Data  Collection  (cont’d) 


►  Selected  Comments  from  Selected  Program  Managers/System  Safety  Leads 

-  TOW  and  JAVELIN  (Army/Marine  Corp),  Mr.  Sam  Taylor  (Marine  Corps);  Principal  for  Safety 

■  For  Submunitions:  Navy  relies  on  “WSESRB  Position  on  Submunitions”  8020  Ser  N71 33/860,  dated 
6  Sept  2000  and  Army  relies  on  “DoD  Policy  on  Submunitions  Reliability"  dated  10  Jan  2001. 

-  Joint  Direct  Attack  Munition  (Navy/Air  Force),  Ms.  Larraine  Hebb  (Air  Force),  System  Safety 
Lead 


■  For  JDAM,  she  specifically  noted  the  HERO  requirements  for  the  Navy  are  often  more  stringent  than 
the  Air  Force’s  requirements;  however,  as  a  rule  the  Air  Force  will  yield  to  the  more  stringent 
requirement. 

-  Lightweight  155  (Army/Marine  Corps),  Gaby  Jarani  (Army),  Principal  for  Safety 

■  The  WSESRB’s  safety  requirements  and  the  Army’s  Material  Release  process  are  completely 
different  approaches. 

►  Identified  and  Collected  Testing  Requirements 

-  Over  75  different  testing  documents  collected  resulting  in  530  distinct  tests 

►  Established  and  Maintaining  the  Explosive  Safety  Testing  Knowledge  Center 
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Data  Analysis  -  Content  of  database  shows  that  some  duplication  of 
safety  tests  exists  when  defined  by  Mode  and  Military  Service 


Identified  Area  of  Duplication  by  Mode  &  Test  Classification 
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0 

0 

Operational  Use 
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0 
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Storage/Stowage 

Transportation 

0 

0 

ALL  UP  ROUNDS/COMPONENTS 

Initial  Results  of  Analysis 


►  Preliminary  analysis  shows  that  duplication  exists,  but  singularities,  commonalities,  and  inconsistencies  have  yet  to  be 
determined. 

►  Although  duplication  has  been  identified,  we  can  not  be  certain  to  what  degree  of  duplication  exists  in  each  mode;  varying  levels 
have  been  established. 

►  Duplication  appears  both  across  different  services  as  well  as  within  a  single  service. 
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Data  Analysis  -  Within  Shock  Testing  a  range  of  duplication  exists, 
from  limited  overlap  to  identifiable  duplication  of  effort 


Cause  &  Effect  diagram  of  “Duplicate”  Shock  Classification  tests 
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Data  Analysis  -  The  level  of  estimated  duplication  for  Shock 
Testing  is  more  evident  in  the  mechanical  short  drop  testing  area 


Level  of  Duplication  by  Short  Drop  Tests 


TESTS 

1.5-M 

Drop 

5-Ft  Drop 
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2 
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4 

2 
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4 

4 

Reload  Drop 

2 

2 

4 

Short  Drop  Results 


►  Duplication  of  existing  tests  appears  to  exist 

-  Causes  for  duplication  are  unknown  as  all  are  Army  tests,  hypothesis  is  due  to  lack  of 
transparency  into  other  program  testing 

-  3-Meter  Drop  and  Reload  drop  are  executed  from  the  exact  same  height 

-  1 .5M  and  5-Ft  tests  also  from  the  same  height 

►  Research  indicates  all  tests  would  be  run  on  the  same  explosive  round  (munitions) 

►  Duplication  at  equal  height  is  apparent  (Reload  Drop  vs.  3M  Drop) 


-  Recommend  further  analysis  on  cost  and  pass/fail  differential  between  3M  and  1 .5M  tests  to 
gauge  the  performance  difference  between  tests 
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Data  Analysis  -  The  level  of  estimated  duplication  for  Shock 
Testing  is  more  evident  in  the  mechanical  short  drop  testing  area 


Level  of  Duplication  by  Long  Drop  Tests 
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High  Drop  Results 


►  Duplication  may  exist  and  should  be  subject  for  further  review 

-  Causes  for  duplication  are  suspected  to  be  due  to  perceived  unique  mission  need  of 
each  test. 

►  Duplication  is  assumed  to  be  between  Army  and  Air  Force  Safety  Tests 

►  Given  the  different  operating  environments  and  modes  identified,  duplication  should  be 
further  explored  but  is  not  expected  to  be  area  of  considerable  inefficiency 
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Data  Analysis  -  Vibration  is  constantly  measured  against  additional 

safety  considerations 

Level  of  Duplication  by  Vibration  Tests 
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►  Lower  levels  of  duplication  exist  than  at  the  Short  Drop  Test,  yet  areas  of  overlap  may  merit  consideration 
for  additional  analysis 

►  Overlap  is  likely  due  to  expansion  of  tests  to  create  and  simulate  extreme  tests  under  a  variety  of 
environments  (hypothesis) 

-  In  testing  extreme  and  variety  of  environments,  elements  of  overlap  emerges 

-  Thermal  Shock  and  Shack  &  Temperature  address  same  elements 

-  Acoustic  Noise  with  Mechanical  Vibration  &  Temp  change  tests  acoustic  noise 
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Agenda 


►  Study  Overview 

►  Methodology 

►  Data  Collection  and  Analysis 

►  Results  and  Conclusions 

►  Next  Steps 
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Results  and  Conclusions 


►  Derived  hypothesis  from  the  defined  problem  and  objectives 

►  Mapped  out  work  plan  and  approach 

►  Defined  database  structure  hierarchy 

►  Built  relational  database  to  capture  analysis 

►  Defined  key  terms  for  the  purpose  of  this  study 

►  Classified  test  modes  and  test  classifications 

►  Collected  comments  from  Service  Board  SMEs 

►  Performed  analysis  on  database  input  and  SME  input 

►  Phase  I  clearly  indicates  that  there  are  potential  savings  to  support  a  Phase  II  effort 
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Agenda 


►  Study  Overview 

►  Methodology 

►  Data  Collection  and  Analysis 

►  Results  and  Conclusions 

►  Next  Steps 
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Next  Steps  -  Phase  II 


►  The  next  phase  will  focus  on  general  agreement  of  the  required  tests  for  each  mode. 

►  Continue  identifying  commonalities  and  differences  between  test  requirements. 

►  Conduct  benefits  and  opportunity  analysis  on  duplicate,  inconsistent  and  unique  test 
requirements. 

►  Provide  recommendations  for  duplicate  and  inconsistent  tests  to  JWSTAP. 

►  Prepare  draft  JWSTAP  document  describing  the  required  tests  by  mode. 
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Agenda 


•  Purpose 

•  Schedule 

•  Scenario 

•  Modeling  and  Simulation 

•  Financial  Analysis 

•  Conclusion 

Supporting  Information  from  previous  IPR  presentations  are  available 
upon  request  -  Scenario,  Flow  and  Entity  Diagrams,  Notional 
Formations,  Detection  Analysis,  SV  Diagrams. 
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Purpose 


•  The  purpose  of  this  presentation  is  to  report 
the  PHD  Cohort  #4  group  analysis  of  the 
Implication  of  FORCEnet  on  Coalition 
Forces. 
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Project  Objectives 


Develop  a  high  level  system  architecture 
to  accomplish  maritime  coalition  littoral 
warfare  operations. 


To  provide  analysis  and  guidance  on 
the  technical  requirements 
for  coalition  maritime  warfare 
interoperability  in  a  FORCEnet  construct. 


•  Assess  the  incremental  value  of  higher  levels  of  FORCEnet 
interoperability  (not  fully  networked  to  1 00%) 

•  If  it  is  determined  that  FORCEnet  is  a  “good  thing’  to  implement 
with  coalition  navies,  then  determine  what  is  the  cost  of  entry  into 
the  USN  version  of  Network-Centric  Warfare. 
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Schedule 


•  Project  Plan 

•  Initial  Capabilities  Document  (ICD) 

•  Architecture  Framework 

•  Modeling  &  Simulation 

•  Financial  Analysis 

•  Capability  Development  Document  S 
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Modeling  and  Simulation 

What  did  we  consider  and  what  were  the  results? 
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Assets  Available 

JT  / 
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Non-Organic  &  Organic  Assets 
Active  /  Passive 

Space  /  Air  /  Surface  /  Sub-Surface 


Model  snapshots 
shown  here  were 
accomplished  using 
AGI  software 


Financial  Analysis 

What  did  we  consider  and  what  were  the  results? 
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Financial  Analysis 
Assumptions 

•  The  dollar  amount  assigned  to  each  element  in  the  cost 
model  is  based  on  information  gathered  from  comparable 
systems. 

•  Procurement  Cost  includes: 

-  System  Integration  Cost  (R&D) 

-  Acquisition  Cost 

-  Installation  Cost 

-  Initial  Training  Cost 

-  Cost  of  Initial  Spares 

-  Administrative  and  Logistics  Cost 

•  Operation  &  Support  Cost  includes: 

-  Ongoing  Technical  Support  Cost 

-  Cost  of  Replenishment  and  Spare  Parts 
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Coalition  FORCEnet 
Fielding  Plans 


•  FORCEnet  Level  I 

-  Spiral  1  Acquisition  (Command  &  Control)  (FY2006) 

•  FORCEnet  Level  II 

-  Spiral  1  Acquisition  (Command  &  Control)  (FY2006) 

-  Spiral  2  Acquisition  (Enhanced  Tactical  Data 
Capability)  (FY2010) 

•  FORCEnet  Level  IV 

-  Spiral  1  Acquisition  (Command  &  Control)  (FY2006) 

-  Spiral  2  Acquisition  (Enhanced  Tactical  Data 
Capability)  (FY2010) 

-  Spiral  3  Acquisition  (Shared  Engagement  Capability) 
(FY2014) 
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Procurement  Costs 


Procurement  Cost  of  FORCEnet  Levels 
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□  Integration 

■  Software 

□  Hardware 

□  Admin  &  Logistics 

■  Installation 

□  Initial  Training 

■  Initial  Spares 

Note:  Costs  are  estimated  based  upon  existing  systems  historical  costs.  Costs  are  per  ship. 
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O&S  Costs 


O&S  Cost  among  FORCEnet  Levels 
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Total  LCC  Costs 


Total  LCC  Cost  among  FORCEnet  Levels 
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Note:  Costs  are  estimated  based  upon  existing  systems  historical  costs.  Costs  are  per  ship  over  20  years. 
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Sensitivity  Analysis 
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Conclusion 
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Conclusion 


•  FORCEnet  Level  4  provides  the  higher 
performance 

•  FORCEnet  Level  4  associated  costs  are 
higher 

•  FORCEnet  Level  2  provides  comparable 
performance  levels  at  ~  40%  less  cost  than 
FORCEnet  Level  4. 
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Purpose 


M&S  frameworks  have  the  potential  to  change 
how  M&S  collaboration  is  performed. 

The  M&S  Framework  and  “Flexible  Syntax” 
developed  at  the  US  Army  Armament 
Research  Development  Engineering  Center 
(ARDEC)  as  part  of  the  Armaments  Server  and 
MATREX  NEC2  projects  has  broad  application 
and  potential  to  become  a  standard  framework 
for  distributed  interoperability  between 
models. 
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Background 


Modeling  Architecture  for  Technology,  Research  &  Experimentation 
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technical,  and  scientific 
support  for  aviation  anC 
missile  weapon  system^ 
and  their  support  ' 

systems,  UAV  platforms, 
robotic  ground  vehicles 

The  principal 
researcher,  developer 
and  sustainer  of 
current  and  future 
armament  and 
munitions  sytems. 

k  develop  and  integrate 
^Dommand,  Control, 
^Communications, 

*  Computers, 

1  Intelligence, 
lurveillance,  and 
Reconnaissance 
(M4ISR)  technologies 
iRt  enable  information 
f  dominance  and 
decisive  lethality  for  the 
networked  Warfighter 

Requirements, 

Design  & 
Development, 
Integration, 

Testing, 

CM, 

Operations 
&  Support 

Provide  life  cycle 
management  of 
interoperable 
training,  testing  and 
simulation  solutions 
for  soldier 
readiness  and  the 
defense  community 

t_ i_ 4_ t_ t 

MATREX 

r 

V 

AMSAA 

ARL 

|  ECBC 

NATICK 

tardec  vja  centralized  data  set 

Conduct 
responsive  and 
effective  materiel 
and  logistics 
systems  analyses 
to  support  decision 
making  for 
equipping  and 
sustaining  the  U.S. 

Army 

Weapons  and 
Materials,  Sensors 
and  Electron 

Devices,  Human 
Research  and 
Engineering, 
Computational  and 
Information 
Sciences,  Vehicle 
Technology,  and 
Survivability  and 
Lethality  Analysis 

*  ECBC  develops 

technology  in  the 

1  areas  of  i 

|  detection,  j 

l  protection,  and  ! 

I  decontamination 

|  and  provides 

i  support  over  the 

jj  entire  lifecycle 

researching, 
developing,  fielding, 
and  managing  food, 
clothing,  shelters, 
airdrop  systems,  and 
soldier  support 
items. 

research,  develop, 
engineer,  leverage 

"advanced'6  □  Allows  i ncorporation  of  additional 

ground'systems  components  that  are  compliant  with  the 

ZIT:  MSDL  format 

throughout  the  life 
cycle 

Benefits: 


□  Reduces  time  and  resources  required 
to  initialize  a  distributed  simulation 

□  Promotes  reuse  of  standardized 
scenario  data  format 
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Project  Goals 


□Develop  an  M&S  framework  that  will  support  ARDEC  M&S 
interoperability  and  distributed  simulation. 

□  Deliver  Effects  on  Target  as  a  vital  piece  of  RDECOM 
MATREX  (Armaments  Server). 

□  Integrated  Process  for  Network  Effects  Command  and  Control 
(NEC2). 

□  Provide  modular  interface  for  both  models  and  distributed 
simulation  network  interfaces. 

□  Minimize  the  impact  of  changes  (from  models,  desired 
behavior). 

□  Use  a  “Flexible  Syntax”  for  configuration  and  rules  files  to. 
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What  Problems  Does  This  Solve? 

□  Allows  disparate  Engineering  models  to 
participate  in  distributed  simulation  exercises 
with  minimal  tailoring  of  the  model. 

□  Enables  model  to  model  interoperability. 

□  Minimize  the  impact  of  changes. 
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Framework  Status 


□  Continuous  Development  Since  Nov  2002 

>  4  contracts 

>  2-3  FTE  Govt. 

>  6  SME  on  and  off. 

□  Accomplishments 

>  Designed  and  Built  Framework 

>  Develop  and  Maintain  Armaments  Server 

s  Used  in  Is*  App. 

✓  Involved  in  MATREX  (Selected  by  FCS  LSI) 

>  NEC2  is  considering  using  the  Framework. 

>  Supporting  the  Development  of  NEC2 

•/  Effects  Engine 

□  Ammunition  Trajectory  Models  Available 

>  PGMM 

>  SADARM 

>  Etc . 
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Architecture  Overview 
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Model  Bus  Server  Architecture 
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HLA  Module  Architecture 
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Bus  Object  Data  Encapsulation 
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What  makes  this  different? 


The  architecture  is  unique  in  that  it 
adapts  to  FOM  changes  without  the 
need  to  recompile. 

>  Most  Federates  need  to  recompile. 

>  This  framework  only  requires  a  changed 
OMT.txt  file  along  with  the  XML  object 
mapper  XML  file 
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Data  Routing  &  Transformation 
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Armaments  Server 


Armament  Server  User  Interface 

- * - 


Event  Monitor 

4 

♦ 

G> 

(/)  "5 
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o 

Configuration  Processor 

t  - 


Decision  Processor 


Model  Bus  Server  Core 
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SIF 
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0 
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>  < 
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>  < 
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)  i 

>  ^ 

c 

f 

r  ' 
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1  SIF  1  |  SIF  2  | 

|  SIF  3  |  SIF  n 

Core  config.xml 


Rules.xml 


Book  of 
Armaments 
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HLA 

TRAJ,  SMAC 

NABK 

CTDB,  GV-List, 

Module 

Module 

Module 

Small  Arms,  Module  n 

SIF  =  Smart  Interface 
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What  Technology  was  used? 


□  Core  Server  Written  in  C++ 

>  uses  standard  template  library 

□  Runs  in  Windows  OS 

>  can  be  ported  to  other  OS’s 

□  Rules  Based  Routing  &  Transformation 
□“Flexible  Syntax” 

□  Server  API 
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What  are  the  Opportunities? 


□Could  be  used  to  develop  other  simulation 
frameworks. 

□Could  be  used  to  replace  existing  simulation 
frameworks  that  are  more  complicated  to 
interface  with  or  maintain. 

□Enable  interoperability  between  models, 
Simulators,  and  hardware 
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Interested  in  Building  a  Server? 


□  Need  API 

>  HPTi  License  agreement  -  unless  govt. 

□  Develop  interface  SW  according  to  API 

□  Generate  Rules  file 

□  Revise  Configuration  file 

□  User  Interface 

□  For  HLA 

>  HLA  module  Licensed  or 

>  Build  your  own 

□  Contact 

>  US  Army  ARDEC  for  govt.  -  govt,  work 

>  HPTi  for  contractor  to  commercial  or 
contractor  to  govt. 
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Questions? 
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NDIA 

9th  Annual 

Systems  Engineering  Conference 


Product-based 
Systems  Engineering 
Requirements 

25  October  2006 

Brian  Shaw,  The  Aerospace  Corporation 
Larry  Pennell,  Sparta,  Inc. 

Dave  Davis,  USAF  SMC/EAE 


THE  AEROSPACE 

CORPORATION 


>  PACTA 


Agenda 

tmmmmmu 

•  Background 

-  Space  System  Acquisition  Challenges 

-  SMC  Specifications  and  Standards  Program 

•  SMC  Approaches  to  Systems  Engineering  (SE) 

-  SE  Processes  and  Requirements  Vision 

-  Development  of  “SE  Products  and  Requirements” 

*  SE  Products  and  Requirements 

-  Sample  Products  and  Requirements 

*  Summary  and  Conclusions 
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•  Space  capabilities  are  essential  for  successful  military 
operations 

-  Requires: 

•  high  reliability 

•  assured  delivery  of  mission  data 

•  Many  new  space  programs  exist;  success  hampered  by 

-  Industry-wide  shortage  of  systems  engineers 

-  Serious  lack  of  acquisition  experience  at  SMC 

•  Congressional  confidence  in  space  systems  acquisition  is  low 

•  Space  programs  are  more  complex  than  ever... 

-  Hardware/software,  including  COTS  infusion 

-  System  integration 

-  System-of-system/Family-of-systems  interoperability 
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Contract: 

•  Defines  SE 
Relationship 

•  Establishes  what 
Govt  considers  to 
be  an  acceptable 
level  of  quality 

•  Levels  playing 
field  in  bidding 
process 

•  Requires 
Government 
control/oversight 


Government 


jJIa  I 
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A&AS 


Prime  Contractor 


Standards  set 
expectations  for 
Contractor 
technical  and 
management 
activities 


Sub-Ktrs 


Engineering 
Interaction  ^ 


Specs  &  Standards 


•  Issued  by  Lt. 
1 1  July  2006 


Gen.  Michael  Hamel 


DEPARTMENT  OF  THE  AIR  FORCE 

nS«XyjAKTE(*S  SPACE  AMO  Ml££Jl£  CfSTELft  C€eftCO.  (AFStCl 
LOS  dM$£i£gr  C-A 


•  Establishes  specifications  and 
standards  as  an  integral  element  of 
SMC  acquisition  processes 

•  Applies  to  all  new  development, 
acquisition  and  sustainment  contracts, 
including  new  contracts  for  legacy 
programs 


JVL  1 1  ZOtf 


MEMORANDUM  FOR  SMC- ALL 
FROM:  SMC/CC 

SUBJECT:  Initial  Policy  on  Specifications  and  Standards  Usage  *1  SMC 

1.  Tliis  polity  establishes  Hie  use  of  specifications  and  standards  as  an  integral  element  of  SMC 
acquisition  processes.  Programs  executed  by  SMC/AFPEO-Space  shall  include  specifications 
and  standards  in  all  solicitations  and  shall  place  them  on  contract  as  compliance  documents 
through  the  supplier  chain,  as  appropriate. 

2.  The  SMC  Chief  Engineer  shall  be  responsible  for  defining,  coordinating,  maintaining, 
updating  and  reporting  the  master  list  of  compliance  documents.  The  list  includes  the  minimum 
essential  government,  industry,,  professional  and  international  specifications  and  standards  for 
SMCl5  total  portfolio  of  launch  vehicles,  space  vehicles,  ground  systems,'  user  equipment, 
missile  systems,  facilities  and  research.  This  policy  applies  to  all  new  SMC/AFPEQ-Space 
development,  acquisition  and  sustainment  contracts,  including  new  contracts  for  legacy 
programs.  For  existing  programs  and  contracts,  the  SPO's,  with  the  SMC  Chief  Engineer,  will 
assess  the  program,  status,  requirements,  technical  baseline  and  risks  to  generate  s  tailored  subset 
of  specifications  and  standards.  This  subset  will  be  recommended  to  SMC/CC/ AFPEO-Space 
for  implementation.  The  necessary  specifications  and  standards  will  be  placed  on  contract,  as 
part  of  the  program's  baseline  and  the  Program  Office  shall  enforce  them.  Any  issues  on 
specifications,  standards  or  implementation  that  arise  between  5MC/EA  and  SPD's  will  be 
brought  forward  to  S MG/C CM F PEO- S fu ce  for  resolution. 

3.  The  Chief  Engineer  shall  prepare  an  SMC  01  to  institutionalize  the  practice  and  intent  of  this 
policy. 


MICHAEL  A.  HAMEL 
Lieutenant  General,  U5AF 
Commander 


•  Contractual  compliance  through  the 
supplier  chain,  as  appropriate 
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SMC  Specs  &  Standards  Initiative 

*  SMC  Specs  &  Standards  Initiative 

-  SMC  initiative  to  apply  specs  &  standards  as  key 
element  of  acquisition  practices  and  toolset 

*  Prior  to  acquisition  reform,  SMC  used  military 
specifications  &  standards  to  specify  contractual  and 
prescriptive  requirements 

*  Re-institute  standards  responsible  for  mission 
assurance 

-  Select  group  of  military  and  industry  space  standards 

-  Define  government’s  expectations  and  specify  proven 
technical  practices 
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•  Revised  SMC  S&S  List  published  8/9/2006: 

65  essential  documents 

-  Military,  international,  and  industry 
standards,  and  Aerospace  Corp  reports 

-  Updated  to  reflect  current  best  practice 

•  Example  standard: 

Systems  Engineering 
Requirements  and  Products 
The  Aerospace  Corporation  Report, 
TOR-2005(8583)-3,  Rev  A 

-  Contractually  binding  requirements 
defined  in  terms  of  required  SE 
products  and  required  attributes  of 
those  products 

-  Consistent  with  existing  industry 
standards  (ANSI/EIA  632  and  IEEE 
1220) 

-  Additional  updates  to  current  document 
versions 
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S&S  Status 

■  ■■■■■III 


Collaboration  Ensures  Consistency 


Co-Chaired  by 
DoD  EA  Space 
&  DNRO 


National  Security 
Space  Integration 

Space  Industrial 
Base  Council 


“Bring  Senior-Level  Attention  to 
Space  Industrial  Base  Issues  on  a  Recurring  Basis 

and 

Develop  ‘Actionable’  Recommendations  to 
Industrial  Base  Issues.  ” 


Specs  &  Standards 
Working  Group 


Ensure  sound  technical  practices  applied  on 
NSS  programs 

Ensure  NSS  community  takes  a  consistent 
approach  in  the  application  of  specs  & 
standards 

SMC;  NSSO;  NRO;  Navy;  NASA;  MDA;  NOAA 
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SMC / NRO 
Collaboration 


Mission  Assurance 
Improvement  Task 
Force 

“Identify  and  implement  areas 
where  a  common  SMC  /  NRO  approach 
provides  benefit.’’ 


Co-Chaired  by 
NRO  DDSE  & 
SMC/EA 


Specs  &  Standards 
Working  Group 

Establish  a  common  set  of  preferred 

specifications  and  standards 

Quarterly  interchange  meetings 

Joint  insight  into  formal  standards  rationale 

and  approval  processes 

SMC;  NRO 


MANAGEMENT 


SMC  Specifications  and  Standards 

Functional  Areas 

TECHNICAL 


•  Configuration  Management 

•  Design  Reviews 

•  Manufacturing  Management  / 
Producibility 

•  Parts  Management 

•  Product  Assurance 

•  Program  Management 

•  Risk  Management 

•  Systems  Engineering 

•  Safety 

•  Subcontract  Management 
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Electrical  Power,  Batteries 
Electrical  Power,  Solar 
EMI  /EMC 

Environmental  Eng  &  Cleanliness 

Human  Factors 

Interoperability 

Logistics 

Maintainability 

Mass  Properties 

Moving  Mechanical  Assemblies 

Ordnance 

Pressure  Vessels 

Parts,  Materials  &  Processes 

Reliability 

Security 

Software  Development 

Structures 

Survivability 

Test,  Ground 

Test,  Space 
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SMC’s  Vision  and  Perspective 

*  Compliance  document  for  contractor 

-  Requirements  for  contractor’s  SE  program 

-  Characteristics  for  technical  areas  important  to  SMC 

-  Industry  standards  for  guidance,  not  direction 

*  Requires  associated  handbook  and  tools 

*  Assumes  a  knowledgeable  and  intelligent  audience 

*  Must  be  priceable  -  concise  without  redundancy 
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SE  Requirements  and  Product  standard  should... 

•  Cover  entire  life  cycle  -  lust  to  dust’ 

•  Be  consistent  with  DAG*,  Mission  Assurance  initiatives,  etc. 

•  Meet  SMC’s  needs  for  enforceability  on  contract 

-  Consist  of  a  specification  plus  a  suite  of  tools  such  as 
handbook,  CDRLs,  and  DIDs 

•  Be  supportive  of  5000.2,  Directive  7,  and  NSS  03-01 

•  Be  systems  engineering  product  oriented 

•  Integrate  and/or  use  any  existing  products,  such  as  those 
developed  under  Mission  Assurance  initiatives 
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Relationship  of  SE  Process  Documents 


SE  Requirements 
and  Products 

Draft 
SE  Standard 

Military 

Standards 

SE  Models  and 
Processes 

DOD 

Acquisition  Policy 


Periodic  revision 
is  anticipated 

Aerospace  TOR 
Revision  A 

Mil-Std  499C  (Draft) 
Aerospace  TOR 


Navy 

SE  Guidebook 


INCOSE 

EIA  632  Revision 


Ever-growing 
Space  Acquisition 
Experience  base 


Mil-Std  499A  Mil-Std  499B  (Draft) 


EIA  632/IEEE  1220 

Defense  Acquisition 
Guidebook 

SMC  SE 
Handbook 

DoDD 

5000.1 

DoDD  5000.2 

NSS  03-01 

THE  AEROSPACE 
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Process  Input 

Customer  Needs/Objectives/Capabilities 
Technology  Base 

Specifications  &  Standards  Requirements 
Requirements  from  Prior  Development  Effort 
Program  Decision  Requirements 


Example  SE  Process 
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Requirements  Analysis 

-  Analyze  Missions,  Concepts  and  Environments 

-  Identify  Functional  Requirements 

-  Define/Refine  Performance 

-  Define/Refine  Performance  Design  Constraints 


Requirements  Loop 


1 


System 
Analysis  & 
Control 


Functional  Analysis  &  Allocation 

_  -  Decompose  to  Lower-Level  Functions 

-  Allocate  Performance  and  Other  Limiting 
Requirements  to  All  Functional  Levels 

-  Define/Refine  Functional  Interfaces  (Internal/External) 

-  Define/Refine/Integrate  Functional  Architecture 


Design  Loop 


Synthesis 

-  Transform  Architectures  (Functional  to  Physical) 

-  Define  Alternative  System  Concepts,  Configuration 
Items  and  System  Elements 

-  Select  Preferred  Product  and  Process  Solutions 
-Define/Refine  Physical  Interfaces  (Internal/External) 


Trade-Off  Studies 
Effectiveness  Analyses 
Design  Analysis 
Risk  Management 
Configuration  Mgt 
Interface  Management 
Data  Management 
Performance  Measurement  j 
Technical  Reviews 
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Process  Output 

Architectures 
Specifications 
^  Engineering  Design 
Decision  Database 
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SE  Process,  Requirement 
and  Baseline  Flow 


Input 

I  Acquisition  Milestone  Decisions  ! 

j  &  Capability  Needs  | 

:  DoDI  5000.2,  NSSAP  03-01,  CJCSI  31 70.01  C  : 

Systems  Engineering  j 

Program  Foundation  j 

(See  Appendix  A) 


System  Analysis  &  Control 
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5.2.2  SUPPORT ABILITY  ANALYSIS  AND  ASSESSMENT 

5.2.3  TRAINING  ANALYSIS  AND  ASSESSMENT 

5.2.4  DISPOSAL  ANALYSIS  AND  ASSESSMENT 

5.2.5  ENVIRONMENTAL  ANALYSIS  AND  IMPACT  ASSESSMENT 
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S^Requimmen^nd Product  Sample 


4.2.3  System  Element  Design  Solution  and  Validation 

The  Contractor  SHALL  determine  the  design  solution,  support  validation  of  the  design  solution,  and 
develop  the  associated^quired  systems  engineering  products  with  the  product  attributes 
specified  in  this  documet 


4.2.3.1  Require* 

a.  The  validated,  approved,  an 
interface  documents  or  their  ele 
segment,  subsystem,  component 

4.2.3.2  Required 

a.  The  allocated  baseline: 

(1 )  Identifies  all  system 

(2)  Documents  the  ass 
and  analyzes  decisic 

(3)  Includes  the  design- 
constraints  for  each 

(4)  Includes  all  derived  d 

(5)  Includes  all  interfaces  a 
the  logical  issues  such  as 

(6)  Includes  the  verification  meth 


tern  Engineering  Products 


jallocf 


ioe  in  specifications  and 


ent  such  as 


Contractor 


Requirement 

Perform  activity; 
produce  product 


criteria; 


/ell  as 
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S^Requimmen^nd Product  Sample 


4.2.3  System  Element  Design  Solution  and  Validation 

The  Contractor  SHALL  determine  the  design  solution,  support  validation  of  the  design  solution,  and 
develop  the  associated  required  systems  engineering  products  with  the  product  attributes 
specified  in  this  document. 

4.2.3.1  Required  System  Engineering  Products 

a.  The  validated,  approved,  and  maintained  allocated  (design-to)  baseline  in  specifications  and 
interface  documents  or  their  electronic  equivalent  grouped  by  each  system  element  such  as 
segment,  subsystem,  component  (hardware  and  software),  computer  software  unit,  and  part 

4.2.3.2  Required  Product  A 


a.  The  allocated  baseline: 

(1)  Identifies  all  system  products 

Documents  the  assessm 
and  analyzes  decision 

Includes  the  design-t 
constraints  for  each 

Includes  all  derived 

Includes  all  interface' 
the  logical  issues  sue' 

Includes  the  verificatio 


(2) 

(3) 

(4) 

(5) 


pision  criteria; 
n 


(6) 
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S^Requimmen^nd Product  Sample 


a 

inte 
segment, 


Product 

content/quality 

metrics 


Solution  and  Validation 

prt  validation  of  the  design  solution,  and 
s  with  the  product  attributes 


ing  Products 

t|n-to)  baseline  in  specifications  and 
£>ed  by  each  system  element  such  as 
are),  computer  software  unit,  and  part 


4.2.3.2  Required  Product  Attributes 

a.  The  allocated  baseline: 

(1 )  Identifies  all  system  products  and  establishes  the  interactions  of  the  system. 

(2)  Documents  the  assessment  of  alternative  solutions;  identifies  and  quantifies  decision  criteria; 
and  analyzes  decision  uncertainties. 

(3)  Includes  the  design-to  technical  functional  and  performance  requirements  and  design 
constraints  for  each  product. 

(4)  Includes  all  derived  design-to  requirements  and  design  constraints  for  each  product. 

(5)  Includes  all  interfaces  and  addresses  how  the  interface  will  be  implemented,  as  well  as 
the  logical  issues  such  as  data  formats,  data  semantics,  etc. 

(6)  Includes  the  verification  method(s)  selected  for  each  requirement. 
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Specialty  Engineering  Requirement  and  Product 


Sample 

■■■■■in 


5.1.13  Reliability 

a.  Space  system  specific  reliability  requirements  are  defined,  allocated,  baselined, 
and  traceable  to  system  requirements. 

(1)  Parameters  and  limits  defined  and  provided  within  the  System  Specification 

(2)  Reliability  requirements  reviewed  against  functional  requirements,  customary 
design  practices 

b.  Applicable  specific  design  tasks  and  analyses  conducted,  including: 

(1)  Failure  Reporting  Analysis,  Corrective  Action  System  (FRACAS) 

(2)  Source  selection  and  vendor  control  procedures 

(3)  Failure  Modes  Effects  and  Criticality  Analysis  (FMECA) 

(4)  Derating  and  margins  of  safety 

(5)  Fault  coverage 

(6)  Single  point  failure 

(7)  Redundancy/single  string 

c.  Reliability  Program  Plan  and  Risk  Management  Plan  developed  for  final  top-level 
space  system 

d.  Items  in  development  that  have  impact  on  support  resources  identified,  including 
time,  people,  money,  parts,  tools,  storage,  and  transportation  assets. 
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Comparison  of  EIA  632 
with  Systems 
Engineering 
Requirements  and 
Products  TOR 


BO*  ?0<r> 
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SE  Requirements  and  Products  and  EIA  632 

,Jm£!l!£8>, 


OEIA  Standard  E1A4V2  PratMMt  tor  Ery—nog  a  Syltom 


I'l  4 ('  ii  1 1 W  iiiiiiwJj  j|  i|  |  |  |i « '!  Mi'Mi  jji  ji  ii  |iTj 


X  X  X  X 


X  X 

X  X 


X  X 

X  XXX 

X  XX 

X  X 

X  X 

XX  X 


CONCLUSION  SE  Requirements  and  Products  TOR  does  not 
preclude  any  significant  632  requirement  In  fact,  it  exceeds 
632  by  MANDATING  specific  process  areas/products  that  can 
significantly  impact  mission  success 


EIA  632 
SE 

Requirements 


•  Systems 
Engineering 
Requirements 
and  Products 

Exceeds 

EIA  632 
Requirements 

•  Basic  SE 
Requirements 

PLUS 

•  Specialty 
Engineering 
Product 
Integration 
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Agenda 

tmmmmmu 

•  Background 

-  Space  System  Acquisition  Challenges 

-  SMC  Specifications  and  Standards  Program 

•  SMC  Approaches  to  Systems  Engineering  (SE) 

-  SE  Processes  and  Requirements  Vision 

-  Development  of  “SE  Products  and  Requirements” 

*  SE  Products  and  Requirements 

-  Sample  Products  and  Requirements 

*  Summary  and  Conclusions 
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Summary 


■■in 


*  Systems  Engineering  is  critical  to  overall  mission  success 

-  Baselines  are  the  product  of  SE  process 

-  Control  of  baselines  is  key  to  SE  success 

*  SE  products  facilitates  SE  verification 

-  Verification  requires  products  with  known  attributes 

*  Contracting  (buying)  SE  must  be  done  differently 

-  Products  must  be  contractually  required  (purchased) 


M 


THE  AEROSPACE 

CORPORATION 


27 


Conclusion 

■  ■■■■■III 

•  Systems  Engineering  (SE)  Requirements  and  Products 

-  Supports  government  and  industry  SE  initiatives 

-  Provides  clear  SE  tasking  and  evaluation  criteria 

-  Being  integrated  with  Mil-Std-1521  C  (Technical  Reviews) 

-  Is  coordinated  with  national  security  space  community 

-  Has  become  the  SE  Compliance  document  for  SMC 
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Back-up  Charts 


Contributors  to  the 
SE  Reauirements  ~  ~ 


•  The  author  acknowledges  the  contributions  of: 

Mr.  Frank  Knight 

•  The  Aerospace  Corporation,  Systems  Director 
Corporate  Chief  Architect/Engineer  Division 

Mr.  Barry  Portner 

•  Innovative  Space  Engineering  Services,  Inc. 

...  and  a  large  number  of  subject  matter  experts  from 
The  Aerospace  Corporation 

•  Systems  Engineering  Division;  Engineering  and  Technology  Division 

•  National  Systems  Division 
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500k 

400k 

300k 

200k 

100k 


Increased  Space  System  Complexity 

Electronic  Parts  / SV  Fit  S/W  SLOC  Count 

1000k 

100k 

10k 


1980  1990  2000 


1975  1995  2015 


*  Increased  complexity  also  results  in: 

-  More  latent  defects  (increased  late  build-cycle  and  orbital  failures) 

-  Greater  test  challenges  related  to  changes  in  technology, 
manufacturing  and  materials 

•  Increasing  electronic  part  or  device  complexity  (e.g.,  ASIC,  FPGA) 

•  Increased  use  and  complexity  of  software 

•  Increased  system  design  and  subsystem  complexity 

•  Lack  of  robust  testing 

-  Ability  and  willingness  to  “test  like  you  fly” 

-  Inclusion  of  off-nominal  test  regimes 
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SMC  Compliance  List 


Compliance  Documents  for  SMC  Acquisitions  dated  08/08/06 


FunctkmaU 
Technical  Area 

Document 

Number 

Title 

Pub 

Date 

Tech. 

POC 

Coramenfe 

CorUgurallon 

Management 

TOR-2M6(i&5B3}-1 

OorflgaraUm  Management 

1-5-Aug-DS 

Donanue.  C 
Aerospace 

Corrtamlrtatlwi 

ASTME  1E4-5-Q3 

ASTThl  E 1 E45.  Standard  =ramce  Id' 
Preparation  cf  Aerospace  Conlamlrallor 
Control  =,lars 

10-May-  EC 

.Ley,  K. 
Aerospace 

MueL  be  tailored  6a  change  Hinfld'to  'Ehal'  and  to  Epeerylhal  PLyer 
InclLdeE  the  U.S.  government. 

Des-lg  r  Rev  ewa 

Ml.  STD -5213  NcdceJ 

"echnCal  Rev  ew&  i  AjoILs  ior  DyEteTiE, 
Eaulcmer:  and  CwnuJer  3  cl  ware 

10-Apr-9E 

Davis,  D 
SMOEAE 

ElBctrical  Powar 

TOR-ZDOS|B5B3J-2 

EJecUfcaJ  Fower  DyslenE,  Direct  Current 
Space  VelUde  Desqn  Requirements 

1 1-May- D5 

.enerlz,  B. 
Aerospace 

ElBctrical  Pcwar 

EKJO-W-E3575A  Rev  ANDtce 

1 

Y,lng  HsTfiBS,  CpaceVeiide  DeElgnAM 
"eEtlrg 

1-Sep-32 

SSmpsor.  M. 
Aerospace 

"altering  available -Item  Mark  Simpson 

ElBctrical  PcwBr,  BattadeB 

TOR-2aD4-;S5E3;r5  RedSlDfl  1 

Space  Salter)-  Dtendard 

1-Apr-C6 

-Iwang,  Vi. 
Aerospace 

ElBctrical  Power.  Go  ar 
CBllB 

At  Mi  5M-11 1  2E05 

Cl  a  fearer  ard  GLalty  ReqL  renente  Itr 
Space-da  fed  Solar  Cells 

15-Aug-C'5 

Reas.  0. 
Aerospace 

ElBctrical  PcwBr,  S-Dlar 
PariBlc 

AIM  Sid  S-1 12  2QE5 

Cl  a  fcator  ard  GLaity  ReqL  ranen:&  Mr 
Spaee-GLa  fed  Solar  PsieIb 

IS-Aug-E'S 

Reas.  0. 
Aerospace 

ElBctromagndlc 

1  ntBiTBrencai' 

Compatibility 

TOR-2aD5;S5E3)-1 

Electromagnet  c  Compatibility  Requirements 
For  Space  Equipment  and  Systems 

E-Aug-35 

Dunbar.  Ml. 
Aerospace 

ElBctromagnetlc 

Inlflirereix:*' 

Compatibility 

Ml  .-STD-461  E 

EJBCtrwnagnes.te  En  Edens  ard  Suecep-.tHIty. 
RequremEnlE  far  the  Cantml  oT 
Electnoniaqnetfc  hteHerarce 

1-Aug-39 

Dunbar.  ML 
Aerospace 

ElBcliomagnetlc 

intarfarerrcai' 

Compatlbllltv 

MII-STD-1S423 

EMC  -Grounding  ReqL  renents  re  Space 
Syetem  -adlllles 

1-IMCV-31 

Dunbar.  M. 
Aerospace 

Environmental  Safety  and 
Occupational  Health 

NAS  411 

HazaroojE  Materials  Management  Program 

1^an-3S 

Caponpcm,  V. 
SMCrAXFY 

Human  Factors 

Ml. -STD-1472- 

DoQ  Ceslgr  Crtlena  Stardard  -  HLman 
Engineering 

26-Aug-E9 

Shaw.  0. 
Aerospace 

G rc unb  segment  hardware,  especlaly  meMleftanEpoilaMe  eyeiem  or 
neltfed  radical  systems.  SeeTQR-5ia32-&5E6-1  for  tailoring 
recorn  merdaUans 

Human  Factors 

ISO  9241 

Ergonomic  Requ  -eTierile  farOTtee  'A'crlt  with 
Visual  Display  unite,  Muiitele  Volumes. 

1936 

Shaw.  0. 
Aerospace 

Grcunflsegmen:  hardware,  eEpeda.ly  rorgroLnd  systems and  circe  Ike 
EhLallcrs 

Human  Factors 

COEUI5  Rev  4.3 

Oomioi  CperaUrg  Enr.lronmeit  ;COEi  User 
Interlace  DpectTcalter  sUIS;,  Version  4.3. 
iCMRsre-ence:  59314) 

15-DhHH 

Shaw.  0. 
Aerospace 

Grcundsegmenr  Eofrware  Lsed  arly  Idr  user  Intehaces. 

Human  Factors 

BMC.'AXE  RptflHMRH-3001-1 

Standard  Fraotce  Human  'Compute-  -iLeTaoe 
Display  Curve ndon &  for  Space  System 
Cperaitens 

14-JaHJI 

Shaw.  0. 
Aerospace 

SMC  greund  segmenr  "T-E.S  used  wily  Tot  Lser  Hlerfas 

Human  Factors 

EIA  FE5-1A 

Electric  Indiefflet  All  lance  Englrear  ng 

Eulet  n  -  human  Engineering  -  =r1ndpe&  and 
Fndlcee.  Ver.  1A 

15-Deo-C'5 

Shaw.  0. 
Aerospace 

Apd  cable  1c  ground  segment  deveocmen:  c.  nteigrallofi  -  equt.-aler:  Id 
DcD  Hardtosk  46E6E 

1  nCBroperatollltyr 
Standardization 

DdDAiCtl  VlJ 

CoD  Architecture  Framework  Vferelon  1.D 

E-Feb-04 

Aw,vad,  4. 
SMC.EA5 

1  ntBroperablllty.1 
Standardization 

DIEFt  36^2.3 

DIDR  Easel  na  Release  ae-20 

27-^un-36 

Awwad.  4. 
SMC-EAS 

DISR  Is  an  mine  tod  rorde,^eocmer:crar  Inltomaton  Tecmoogy  i ;rj 
EUnoands  wofle  iRepaced  JclrtTechn^LAjciHedure.'i 

Logistics 

Ml  _-=R.--49EC6 

Logl&llss  Managenarr  Interm  alter  irepases 
13B6-1AJ 

11-40.-95 

Duphtly,  R 
Aero-space 

Logistics 

ML -STD-1 30  M 

Identification  Marking  of  U.S.  Miliary  Fro-parly 

2-Dec-35 

Ward  land,  G 
SMCiLGA 
Duphllty,  R 
Aerospace 

S-MC  Compliance  UeLS  Aug  2GC6 
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SMC  Compliance  List 


Fmcficmal/ 

Duran  meirtt 

Title 

Pub 

Tech 

Comments 

Technical  Area 

Number 

Date 

ROC 

Logistics 

M  _-STO-13E7A 

Packaging,  Handing.  Storage,  ard 
“raoBForatlllty  FrograT  Req  Lire  me  res  for 
GvcLeTsanc  EcupTents 

1-Oct-E>9 

Duphlly,  R 
Aerospace 

Logistics 

TM-9&-D1 

Air  Faroe  “ethnical  Mar  jal  Ccnrracl 

Recu  TSTieTlG  iTMCR; 

1-JLn-97 

Duphlly.  R 
Aerospace 

altering -available  Trcmi  SUCrtJGA 

Logistics 

M  _-=R“-29E12B 

“raining  Data  = rod  l  cos 

31-Aug-DI 

Duphlly.  R 
Aerospace 

“sllcrlngavallaHeirom  SMDi'.iDA. 

Maintainability 

MI_-5TD-4.73B 

MafitSnaKIdy  Frograrr  'or  Systems- ana 
Equlpmer: 

1-JLIT-9E 

Duphlly.  R 
Aerospace 

r-1  a  n  i j  r  n  c :  l  r  1  n  c 
Management.1 

Produc-Id  liltv 

MIL-STD-152EA 

Frcducdcn  Managemenl 

l-Sep-96 

Oavk 

SMC.EAE 

Mas*  Properties 

TOFt-saza  isesovsstd 

Mass  Frcpenee  Comrol  Stan  Hard  tor  Space 
VeflkSes 

HKUI-D5 

Yang,  L. 
Aerospace 

Moving  Maclnanleal 

ABsanbllea 

AIAA  3-1 14-2D05 

Mov  ng  Mechanics  AseencleE  tor  Space  and 
Lauren  Vehicles 

3<Kun-3S 

Gere,  3. 
Aerospace 

Cfdna  oca 

AIM.  3-1 13-2EOE 

Criteria  for  Explosive  Systems  and  Devices 
Lises  or  Space  and *unch  Vehicles 

10-4OV-C<5 

■Scls&ten,  s. 
Aerospace. 

Parts  Management 

ANSI  .'AIAA  R-1QDA 

Recommended  ^racllce  tor  Parts 

Management 

16-Aug-DI 

Davis,  D 
SMC.'A^E 

Apcbatle  1c  ground  syelems 

Parts  Management 

TOFt-2aD4{™g)-3315  Rev.  A 

Fats.  Materials,  i.  >ocesEesGonLrcl 

Fraq-am  tor  Space  VeNcles -  ReMlsor  A 

12-Aug-C^ 

Rocetson,  S. 
Aerospace 

Rep  acemert  TOR  fTOR-aaa&;85E3>-E2S5]  snlbpated  to  te  pjcllMS-d  In 
AUCLSO  2M6 

Parts  Management 

TOR-2aD+;59KJ-M13  Rev.  A 

Technical  Recu  TemeUE  tor  EleclrarioPan&1 
Malelals,  and  Processes  Lsed  lr  -Spas 
Vehicles  -  Rev  son  A 

12-Aug-C^ 

Rocertsom,  5. 
Aerospace 

RepacemertTOR[TTOR-2DD6|B5B3>-5236]  smiepated  to  te  pjclls-ne^d  In 
AjugLs:  2  DOE- 

Parts  Management 

TOR-&3|  14 12  VI 

Rev  A 

Fats.  Materials,  and  Rpceesefi  Conlrcl 
Frcg-am  tor  Expemoscle  .aunct  Vehicles  - 
Revls-on  A 

l-Jan-04 

Gharg.  C-.  E. 
Aerospace 

PrsasurlzBC  Hardwam 

AIAA  S-D8Q-1 999 

Space  Sys:ems,  Mela  lie  Fressuto  Vesses 
FresELlzed  Structures,  and  nasaira 

Oa  reporters 

l-Sep-36 

Gharg,  J.B. 
Aerospace 

PrsBsurlzBd  Hardwara 

AIAA  S-D81A-2D9D 

A  AA  Standard  tor  Space  Systems  — 
Composite  Overlapped  Piessum  Vessels 
(CCPVE) 

1-Dec-DD 

Gharg,  J.B. 
Aerospace 

PrsasurtzBd  HardwarB 

TOR-SaUo  I3ES3V2396 

Space  SySenre  ~  =llgh:  Fressclzed  Systems 

31-Aug-D3 

Gharg,  J  B. 
Aerospace 

PrsBsurlzBd  HardwarB 

TOR-2aDo:&3E3J-233S  Rev.  1 

Sdld  Roclcet  Motor  Case  Design  &  Tesl 
RequremefTlE 

22-Dec-E^ 

Gharg,  J.B. 
Aerospace 

Product  Assurance 

SAE  AS91CO  R=v.  5 

GLasty  DyslenE  -  Aerospace  -  Model  tor 

GLa  ity  ABSurance  In  Design,  D  eve  io  pine  nl. 
Frcdjcdcn.  Inscalladon  and  Ser.  clrg 

-id  1-0= 

Davis.,  D. 
SMC.EAE 

Program  MaragBmeol 

ISO  1459C-2 

Space  Syscems  Frcg-amme  Management  - 
Fat  2:  ProdLct  Assurance  -  Fsicy  £ 

Frhcples 

l-Jlil-32 

Davis,  D. 
SMC.EAE 

Program  Maragemenl 

ISO  1 4oDC-1 

Space  Systems  -  Program  Management  -  Par 
■  Struciulrg  o'  a  programme 

1-Deo-32 

Davls^  D 
SMG.EAE 

Program  Managamenl 

El  A  743 

Earned  Value  Managemenl  Systems 

2'3-Aug-[2 

Davls^  D 
5MG.EAE 

Reliability  Program 

MIL-STD-1S433 

Relacllry  Pregram  Requirements  'or  Space 
and  LaLtcn  VeNcles. 

1-Gct-ES 

Sctilpper,  G 
Aerospace 

Reliability  Program 

M  --STD-78EB.  hallCEE  1  i  2 

Relladtov  Prcg-am  tor  Systems  amc 

Eqjlcmer:  Development  and  Prcskjcllcn 

1-Aug-9E. 

•Sctilpper,  G 
Aerospace 

Ap cl  c-a tie  1c  ground  syelems 

Risk  Management 

ISO  1 76E6 

Space  Systems  -  Rlsfc  Management 

1 -Apr- Co 

Dar.  R. 
Aerospace 

Safety.  Range 

AFBFCMAN  3 1-71 D 

Rarge  Satecy  User  ReqUremenlE  Manual 

l-JLl-34- 

iuarg,  L. 
■SMCi'SE 

Guceme-ded  EWR  127-1 

SMC  Gcmpliarc*  Lists. Aug  2X6 
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Fmctnna V 
Technical  Area 

Document 

Number 

Title 

Pub 

Date 

Tech 

PGC 

Comments 

Safety,  System 

WI.-ST3-E32C 

Syiterr  Safer*1  Frog-am  Requirements 

1^an-33 

iuarg,  L. 
SMCffiE 

Version  D  Is  rro&l  CLner::  SMS' EE  requires  Version  C  Id  ce  ussd  on  SMS 
cordractE. 

Sated  1e  Clsposa 

TQR-0336  (35633-4474 

Recu  renterm  tor  =nd-of-_re  Dl&cosa  nr 
SarellteB  Cparadrq  at  Geosynchronous 
AIBlLde 

3- Wo- 3  5 

Alor.W. 

Aerospace 

Secur  :y 

COD  351D.1-M 

DoD  In'oriradar  Teshroogy  p"j  Security 
Oenricalten  and  .AccrEdfadon  ;C£A;<  ^raoesE 
(MTSCAPJ  Applcaoon  Manua 

31-.UFD0 

OupUs,  J. 
Aerospace 

“altering  :c  gerera1eneqLtremer:5  targuagea^allatieltom  BMC.'^IF 

■Security 

CoC  BSML2 

Ir'crmartlar  Assurance  Implemsnladofi 

M-“ed-33 

3UCL  >.  J. 
Aerospace 
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Stppcrtlo  AoqLlsJdon  Pregram  Protocflwi 

10-Sep^97 

3UCL  &.  J. 
Aerospace 

“altering  :c  gerera1e  reqL  rener:&  language  ax'allatle  team  EIhKPIP 

Security 

WI_-iDHK  173EA 

Syiterr  SacLffly  Engheer  ng  Program 
Management  REdulroTenls 

1-AJUg-35 

Gucl  e.  J. 
Aerospace 

“altering  :s  gereraleneqUrenveriE  language  a^allatie  team  St^C.'^IF 

■Security 

CCD  S3  Manual 

FroteolIngSenstve  Compartenened 
In'ormariSar  VVItelr  Intormadon  Systems 

11-3ec-CG 

3UCL  S.  J. 
Aerospace 

“altering  reqUred  :c  gererale  csr:rad:dr  recu  renienlB  tor  portions  o' tee 
EVEleni  proresE  no  SC  .  Avalable  toom  SMG'FP 

-Security 

T5R3 

“ateoonimurlca:lar&SECLrt1y  Requirements 
Da  cum  Hit 

lATiden  tor 

eaon 

application 

DupLte,  J. 
Aerospace 

NBA-ptcclired.  system  Epecfflc  dacLmerd  that  speeffle*,  requlrerrenls ter 
cryptography  and  hey  rranagemer: 

Software  CevelDpir^nl 

ISO.'IEC  STD  15933 

Solwars  engineering  -  Sofflwara 

Measurement  Process 

11-JU-02 

Zarrbrana.  M. 
Sh/CiEAB 

S-cTt^are  CevelDprrBrri 

RTCA- DO-27  E 

GLldel  nas  ForCommurlcailan.  Ma^gadm, 
Sumeillanee  And  Air  “raffle  ManageTent 
(CNS.'ATMj  Systems  Soltoare  totegrty 

Asel  ranee  -  DO-273 

E-Mar-02 

Zarrbrana.  M. 
SfcyOEAS 

Applicable  to  AIRBORNE  &y5lem5  only. 

Software  CevelDprr^ril 

RTCA-DO-17EB 

Sen  ware  ConsdariUcHE  In  Airborne  Systems 
and  EqUpmenl  CerltTteadDn 

1-Dec-32 
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Concept  for  an  Enterprise  Wide  (EW) 
System  Engineering  (SE) 
Collaborative  Engineering  Environment  (CEE) 


Dr.  David  Signori 
Dr.  Jimmie  McEver 
Dr.  Kenneth  Jordan 
Dr.  Stuart  Starr 


October  25,  2006 


•  Selected  Definitions 

•  Key  Questions 

•  Strategy 

•  Summary 


•  SoS  (source:  B.  E.  White) 

-  “A  collection  of  systems  that  functions  to  achieve 
a  purpose  not  generally  achievable  by  the 
individual  systems  acting  independently” 

•  SoSE  (source:  J.  E.  Kaplan) 

-  “The  cross-system  and  cross-community  process 

-  That  ensures  the  development  and  evolution  of 
mission-oriented  capabilities 

-To  meet  multiple  stakeholders’  evolving  needs 

-  Across  periods  of  time  that  exceed  the  lifetimes  of 
individual  systems” 


Joint  Service 
Capabilities 
(JCIDS  FCBs) 

Faster  individual  system 

evolution  and  4  t 

better  individual  system 
optimization,. ..but  may 
lead  to  problems  in  overall 
interoperability  and 
integration 


Individual 

Systems 


Service 

Functions 

Collection 

[e.g.,  FCJ 


The  GIG 
Enterprise 
[Our  Focus] 


Better  overall 
Interoperability  and 
Integration,... 
but,  may  lead  to 
individual  system 
sub-optimization;  not 
well  matched  to 
existing  processes 


•  An  integrated,  scalable,  fully  distributed  processing  and 
transport  environment,  commercial-technology  based,  that: 

-  Moves  information  from  any  source  to  any  destination 

-  Provides  tailored  information  through  intelligent  pull 

-  Is  dynamic,  adaptive,  self  reconfiguring,  robust,  and  secure 

-  Integrates  legacy  C4ISR  systems 

-  Permits  full  exploitation  of  sensor,  weapon,  and  platform 
capabilities 

•  Joint  cooperative  component 

•  Sensor-to-sensor  for  cueing 


Source:  Mike  Frankel,  former  DASD(C3I  Systems) 


A  Characterization  of  the  GIG  Vision  ^ 


Agenda 


•  Key  Definitions 
^Key  Questions 

•  Strategy 

•  Summary 


The  DoD’s  3  major  decision-making  processes  are  not  structured 
to  support  cross-cutting,  department-wide  development  efforts 
such  as  the  GIG 

-  JCIDS 

•  The  new  process  is  not  yet  identifying  shortfalls  and  gaps 
in  joint  military  capabilities  on  a  department-wide  basis 

•  Requirements  setting  continues  to  be  driven  by  Service 
perspectives 

-  PPBES 

•  Is  structured  in  terms  of  individual  Service  programs  and 
outdated  mission  areas  instead  of  cross-cutting  capabilities 
such  as  net-centricity 

•  It  is  not  flexible  enough  to  quickly  accommodate 
requirements  resulting  from  lessons  learned  or  from  rapidly 
emerging  technologies 


-  Acquisition 

•  Is  unsuited  to  developing  a  system  of 
interdependent  systems  such  as  the  GIG 

•  DoD  has  struggled  to  achieve  Service  buy-in  on 
joint  Service  development  programs  to  address 
interoperability  problems 

-  Overall 

•  The  lack  of  integration  among  these  three 
processes  makes  it  difficult  to  ensure  that 
development  efforts  are  affordable  and  technically 
feasible 


•  A  robustly  networked  force  improves 

-  Information  sharing 

•  Information  sharing  and  collaboration  enhances 

-  Quality  of  information 

-  Shared  situational  awareness 

•  Shared  situational  awareness  enables 

-  Collaboration 

-  Self-synchronization 

Bottom  Line:  Dramatic  increase  in  mission  effectiveness 


Key  Questions 


•  Can  we  adapt  the  tenets  of  net-centricity 
to  guide  SoSE  for  the  GIG? 

•  If  so,  what  Enterprise  Wide  (EW),  System 
Engineering  (SE)  Collaborative 
Engineering  Environment  (CEE)  would  be 
needed  to  implement  that  concept? 


•  Selected  Definitions 

•  Key  Questions 
;  Strategy 

•  Summary 


The  Collaborative  Engineering 
Environment  (CEE) 


•  CEE  --  features  and  purpose 

-  An  SE  force  networked  together  and  with  stakeholders 

-  Supported  by  shared  tools,  services  and  information 

-  That  empower  various  communities  of  interest  to  solve  a 
range  of  problems  collaboratively 

•  Key  attributes 

-  Empower  culture  change,  guidance  and  management 

-  Provide  repositories  and  websites  for  posting  and  sharing 
information 

-  Support  continual  assessment  and  decision  making 
through 

•  Analytical  tools  and  services 

•  Integrated  M&S  environments 

•  Distributed  test  beds 


•  Accessed  through  an  easy-to-use  web  portal 

•  Navigational  aids  to  help  users  find  the 
information  they  need 

•  Incentives  to  provide  up-to-date  information 

•  Designated  Points  of  Contact  (POCs)  to  feed  the 
CEE 


FAQs  and  help  desk 


Empowering  Culture  Change, 
Guidance  and  Management 


♦ 


•  The  CEE  should  serve  to  empower  culture 

change,  guidance  and  management  by  providing 

-Access  to  education  resources 

-Top  level  guidance  for  end-to-end  (E2E) 
capabilities 

-  POCs  for  all  levels  of  management  and 
systems  -  responsibility  matrix 

-Access  to  latest  products  (e.g.,  NCIDs,  JCIDS 
products) 


Repositories  and  Websites  for 
Posting  and  Sharing  Information 


❖ 


•  The  CEE  should  provide  access  to  the  following 

-  Enterprise  and  individual  program  schedules 

-  Enterprise-wide  standards 

-  Current  performance  and  interface  issues 

-  System  emulations  to  support  interface 
development  and  testing  (supplied  and 
updated  by  programs) 

-  Clearing  house  for  suggestions  for  E2E 
system  improvement 


Elements  of  a  CEE  Tool  Box: 
“Soft”  Tools 


♦ 


•  These  tools  should  be  useful  in  rapidly 

-  Performing  exploratory  analyses 

-  Identifying  the  “interesting”  parts  of  solution  space 

•  Representative  “soft”  tools  include 

-  Expert  elicitation  techniques;  e.g., 

•  RAND’s  Subjective  Transfer  Function  approach 

•  Polling  techniques 

•  Multi-attribute  Utility  Theory  (e.g.,  Analytical 
Hierarchy  Procedure) 

-  System  dynamics  models  (e.g.,  CAPE) 

-  Simple,  agent  based  models  (e.g.,  MANA) 


Elements  of  a  CEE  Tool  Box: 
Constructive  M&S 


•  These  constructive  M&S  tools  should  be  used  to 

-  Build  on  the  results  of  “soft”  tools  to  support  more 
detailed,  quantitative  analyses 

-  Provide  baseline  results  to 

•  Guide  the  conduct  of  testbed  assessments 

•  Extend  testbed  results 

•  Representative  constructive  M&S  include 

-  OPNET/NETWARS  (transport  layer  analyses) 

-  NESTOR  (service  layer  analyses) 

-  JWARS  (military  mission  analyses) 


Elements  of  a  CEE  Tool  Box: 
Testbeds 


It  is  anticipated  that  these  testbeds  will 

-  Support  the  more  credible  assessment  of  key  Measures  of 
Merit  (drawing  on  the  insights  developed  from  constructive 
M&S) 


-  Facilitate  the  harmonization  of  the  GIG  with  Service  visions 
(e.g.,  LandWarNet,  FORCEnet,  Constellation  Net) 

-  Provide  insights  to  enhance  the  quality  of  existing 
constructive  M&S 


•  A  variety  of  testbeds  are  envisioned 

-  System-of-systems  testbeds  (e.g.,  N RL-orchestrated 
testbeds) 

-  Mission-oriented  testbeds;  e.g., 

•  FCS  SoSIL  (USA) 

•  Distributed  Mission  Operations  Center  (DMOC)  (USAF) 


Key  Testbeds  to  Exploit 


Extensive  Industry  &  Service 
Participation  in  all  Venues 


Airborne  Network 
Testbed 


GIG  End-to-End 
Testbed  (NRL) 

A  Loaded  and  Stressed  Network ... 
Emulates  “War  of  the  Future”! 


JTRS-TSAT-GIG 
connectivity  example 


TSAT  Tested  (MIT/LL) 

*  Optical  Comm  Testbed 

Testbed  ) 

•  NetWork-Fgstb&d*'* 


Provides  inter-network  demonstration  and  validation 


Strategy:  Use  of  the  CEE 
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•  Become  knowledgeable  about  NATO  Code 
of  Best  Practice  (COBP)  for  C2  Assessment 

•  Apply  Net-Centric  Data  Strategy  to  guide 
the  collection,  conversion,  storage  of  data 
generated  through  application  of  the  CEE 
tool  box 

•  Develop  experimental  design  skills  that 
enable  you  to  apply  the  designated  models 
efficiently  and  effectively 


•  Selected  Definitions 

•  Key  Questions 

•  Strategy 
^Summary 


•  A  CEE  may  enable  a  net-centric  approach  to  SoSE  for  the 
GIG 

•  The  CEE  should  serve  to 

-  Empower  culture  change,  guidance  and  management 

-  Provide  repositories  and  websites  for  posting  and 
sharing  information 

-  Support  continual  assessment  and  decision  making 

•  If  the  CEE  is  to  be  viable,  it  must  be 

-  Evolvable  and  scalable  to  be  consistent  with  the  nature 
of  the  GIG 

-  Consistent  with  the  Services’  plans  to  create  and  apply 
CEEs  to  address  their  own  SoSE  issues  (e.g., 
FORCEnet,  LandWarNet,  ConstellationNet) 


Performance-  Based 
Earned  Value® 


NDIA  Systems  Engineering  Paul  J.  Solomon,  PMP 

Conference  Performance-Based  Earned  Value® 

San  Diego,  CA  Paul.Solomon@PB-EV.com 

October  25,  2006 


©  2006  Paul  J.  Solomon 
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Agenda 


•  DoD  Policy  and  Guidance,  Customer  Expectations 

•  Standards,  Models,  and  Best  Practices 

*  Project  Management  with  Performance-Based  Earned 
Value®  (PBEVSM) 

*  Better  Acquisition  Management 


©  2006  Paul  J.  Solomon 
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Copyright  Attribution 


Copyright©  2006  by  Paul  Solomon. 

This  material  contains  excerpts  from  the  book, 
Performance-Based  Earned  Value.® 

The  excerpts  were  reprinted  courtesy  of  John  Wiley 
&  Sons,  Inc.  and  the  IEEE  Computer  Society  Press. 
Performance-Based  Earned  Value®  was  written  by 
Paul  Solomon  and  Ralph  Young  and  copyrighted  by 
the  IEEE,  2007. 
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Project  Management 
Shortfalls 


*  Inadequate  early  warning 

*  Schedules,  EV  overstate  true  progress 

*  Remaining  work  underestimated 


©  2006  Paul  J.  Solomon 
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Does  EVMS  Really  I  ntegrate? 


COST 


TECHNICAL 

PERFORMANCE 


EVMS 

A 

4 

4 

A 

A 

A _ A 

SCHEDULE 


RISK 


Risk  Profile 


100 

1  Hi  i  i  i  I 


©  2006  Paul  J.  Solomon 
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Value  of  Earned  Value 


7 

7 

Z 

EVM  data  will  be  reliable  and  accurate 
only  if: 

•  The  right  base  measures  of  technical 
performance  are  selected 

and 

•  Progress  is  objectively  assessed. 

\ 

©  2006  Paul  J.  Solomon 
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DoD  Policy  and  Guidance, 
Customer  Expectations 


©  2006  Paul  J.  Solomon 


Government  Pays  But  Fails  to 
Get  Desired  Outcomes 


GAO 

Report 

Title 

Findings  and  Recommendations 

06-66 

Defense 
Acquisitions: 
DOD  Paid 
Billions  in 

Award  and 
Incentive  Fees 
Regardless  of 
Acquisition 
Outcomes 

•Contractors  not  held  accountable 
for  achieving  desired  outcomes: 
o  Cost  goals 
o  Schedule  goals 
o  Desired  capabilities 
•  Programs  do  not  capture  early  on 
the  requisite  knowledge  needed  to 
effectively  manage  program  risks 

06-391 

Defense 
Acquisitions: 
Assessments 
of  Major 
Programs 

DOD  needs  to  change  its 
requirements  and  budgeting 
processes  to  get  desired  outcomes 
from  the  acquisition  process 

(a)  Government  Accountability  Office 

©  2006  Paul  J.  Solomon 
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GAO  Best  Practices 


GAO 

Report 

Title 

Findings  and  Recommendations 

04-722 

Information 

Best  Practices  and  Controls: 

Technology: 

•  Ensure  that  requirements  are 

DOOs  Acquisition 

traceable,  verifiable,  and  controlled. 

Policies  and 

•  Trace  requirements  to  system  design 

Guidance 

specifications  and  testing  documents. 

•  Continually  measure  an  acquisition’s 

06-215 

DOD  Systems 

performance,  cost,  and  schedule 

Modernization 

against  approved  baselines. 

©  2006  Paul  J.  Solomon 
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DOD  Policy  &  Guidance  on  SE 


Policy  for  Systems  Engineering  in  DOD  Policy  2/20/04 


Defense  Acquisition  Guidebook  (DAG)  10/8/04 


Systems  Engineering  Plan  Preparation  Guide  (SEP) 

2/10/06 


WBS  Handbook,  MN-HDBK-881A  (WBS)  7/30/05 


Integrated  Master  Plan  (IMP)  &  Integrated  Master 
Schedule  Preparation  &  Use  Guide  (IMS)  10/21/05 

Risk  Management  Guide  for  DOD  Acquisition  (RISK) 

Aug.  06 


©  2006  Paul  J.  Solomon 
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DOD  Policy  &  Guides 


Policy  or  Guide  (1  of  3) 

Policy 

DAG 

SEP 

WBS 

IMS 

Develop  SEP 

P 

4.2.3.2 

1.0 

Technical  reviews: 

•  Event-driven  timing 

P 

4.5.1 

3.4.4 

3.2.3.1 

2.3,  3.3.2 

•  Success  criteria 

P 

4.5.1 

3.4.4 

3.2.3.1 

•  Assess  technical 
maturity 

4.5.1 

3.4.4 

3.2.3.1 

Integrate  SEP  with: 

•  IMP 

4.5.1 

3.4.5 

1.2,  2.3 

•  IMS 

4.5.1 

3.4.5 

1.2,  2.3 

•  Technical  Performance 
Measures  (TPM) 

•  EVM 

4.5.1 

3.4.4 

3.4.5 

1.2,  2.3 
1.2,  2.3 

©  2006  Paul  J.  Solomon 
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DOD  Guides 


Guide  (2  of  3) 

DAG 

SEP 

WBS 

IMS 

Integrate  WBS  with  requirements 
specification,  statements  of  work 
(SOW),  IMP,  IMS,  and  EVMS 

2.2.3, 

3.2.3.3 

3.4.3 

TPMs  to  compare  actual  vs.  plan: 

•  Technical  development 

•  Design  maturity 

4.5.5 

3.4.4 

3.3.2 

TPMs  to  report  degree  to  which 
system  requirements  are  met: 

•  Performance 

•  Cost 

•  Schedule 

4.5.5 

3.4.4 

Standards  and  models  to  apply  SE 

4.2.2 

4.2.2.1 

Institute  requirements  management 
and  traceability 

4.2. 3.4 

3.4.4 

©  2006  Paul  J.  Solomon 
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Technical 


Standards,  Models,  and  Best 
Practices 


©  2006  Paul  J.  Solomon 
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DOD  Technical  Baselines 


DAG  Technical 

Review 

DAG  Baseline 

DAG 

IEEE  1220 

System 

Functional 

Review 

System 

Functional 

Baseline 

4.3. 3.4.3 

Validated 

Requirements 

Baseline 

Preliminary 
Design  Review 

System 

Allocated 

Baseline 

4.3. 3.4.4 

Verified 

Physical 

Architecture 

Critical  Design 
Review 

System  Product 
Baseline 

4.3. 3.4.5 

Verified 

Physical 

Architecture 

Production 

Readiness 

Review 

System  Product 
Baseline 

4.3.3.9.3 

Verified 

Physical 

Architecture 

©  2006  Paul  J.  Solomon 
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Requirements  Progress 


IEEE  1220 

EIA-632 

6.8. 1.5  Performance-based 
progress  measurement 

6.8.6  Track  product ...  metrics 

4.2.1  Planning  process, 

Req.  10:  Progress  against 
requirements 

6.8.1. 5  d)  Assess 
•Development  maturity  to  date 
•Product’s  ability  to  satisfy 
requirements 

6.8.6  Product  metrics... at  pre- 
established  control  points 
enable: 

•  Overall  system  quality 
evaluation 

•  Comparison  to  planned  goals 
and  targets 

Assess  progress  . . . 

•  Compare  system 
definition 

Against  requirements 
a)  Identify  product  metrics 
and  expected  values 

■  Quality  of  product 

■  Progress  towards 
satisfying  requirements 

D)  Compare  results  against 
requirements 

©  2006  Paul  J.  Solomon 
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Technical  Performance 
Measures  (TPM) 


IEEE  1220:  6.8.1. 5. 

EIA-632:  Glossary 

Performance-based  progress 
measurement 

TPMs  are  key  to  progressively 
assess  technical  progress 

Predict  future  value  of  key 
technical  parameters  of  the 
end  system  based  on  current 
assessments 

•Establish  dates  for 

-  Checking  Progress 

-  Meeting  full  conformance 
to  requirements 

Planned  value  profile  is  time- 
phased  achievement 
projected 

•  Achievement  to  date 

•  Technical  milestone  where 
TPM  evaluation  is  reported 

©  2006  Paul  J.  Solomon 


16 


*  How  well  a  system  is  achieving  performance 
requirements 

*  Use  actual  or  predicted  values  from: 

-  Engineering  measurements 

-  Tests 

-  Experiments 

-  Prototypes 

*  Examples: 

-  Payload 

-  Response  time 

-  Range 

-  Power 

-  Weight 


©  2006  Paul  J.  Solomon 
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Percent  Required  Value 


TPM 


Use  TPMs  as  a  base  measure  of  EV 


©  2006  Paul  J.  Solomon 
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Success  Criteria  of 
Technical  Reviews 


IEEE  1220,  Preliminary  design  stage 
5.2.4.1  Subsystem  reviews 
a.  Subsystem  definition 
*  Mature 

-  Meet  SE  milestone  criteria 

a.  Component  allocations  and  specifications 

-  Provide  a  sound  subsystem  concept 

c.  Subsystem  risks  assessed  and  mitigated 

d.  Trade-study  data.. .substantiate  that 
subsystem  requirements  are  achievable 
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Synthesis  (Design) 


IEEE  1220,  (6.6):  Success  Criteria 
*  Design  solution  meets: 

-  Allocated  performance  requirements 

-  Functional  performance  requirements 

-  Interface  requirements 

-  Workload  limitations 

-  Constraints 

-  Use  models  and/or  prototypes  to  determine 
success 
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Product  Requirements 


CMMI,  PMBOK  Guide:  Traceability  and  consistency 

Requirements  Work 


•Project  Plans 
Task  1  A- A 

>Task  2  A- A 

Task  3  A  A 

•Activities 
•Work  Products 
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Process  and  Product  QA 


•  Product  QA 

•  CMMI: 

Objectively  evaluate  work  products 
against  clearly  stated  criteria 

•  Minimize  subjectivity 

•  EVMS: 

•  EV  is  measurement  of  quantity  of  work 

•  “Quality  and  technical  content  of  work 
performed  are  controlled  by  other  means!” 
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Project  Management  with 
Performance- Based  Earned 
Value®  (PBEVSM) 
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PBEV 


*  4  Principles  and  16  Guidelines 

*  Specify  most  effective  measures  of  project 
performance 

*  Requirements-driven  plan 

*  Consistent  with  standards  and  models 

*  Tailorable  and  scalable,  depending  on  risk 

*  Lean 
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PBEV  Based  on  Standards 

and  Models 


•  ANSI/EIA-632 

•  IEEE  1220 

•  CMMI® 

•  PMBOK®  Guide 

•  INCOSE  SE  Handbook 

•  PSM.  Practical  Software  and  Systems  Measurement:  A 
Foundation  for  Objective  Project  Management 

•  Earned  Value  Management  Systems  (ANSI/EIA-748-A- 
1998,  reaffirmed  August  28,  2002)  (EVMS) 
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Principles  of  PBEV 


1.  Integrate  product  requirements  and  quality  into 
the  project  plan. 

2.  Specify  performance  towards  meeting  product 
requirements,  including  planned  quality,  as  a 
base  measure  of  earned  value. 

3.  Integrate  risk  management  with  EVM. 

4.  Tailor  the  application  of  PBEV  according  to  the 
risk. 
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Supplemental  PBEV  Process  Flow 


Define  the  work 
(WBS) 


(P)  Establish  product  Guideline  1 .1 
requirements  and 
components 
(technical  baseline) 


Plan  the  work 
(Schedule  &  Budget) 


H 


(P)  Integrate  product  Guidelines  1.2,  2.2 
requirements  and 

quality  with  plan 


Execute  the  plan 


(P)  I  ntegrate  risk 


Guidelines  3.2,  3.2, 


management  with  plan  4.1,  4.2 


Incorporate 

internal/external 

|  Analyze  variances  j 

changes 

(P)  Measure  progress 
towards  meeting  product 
requirements  and  quality 

Guideline  2.7 


(P)  =  Supplemental  PBEV  Process 


Implement 
corrective  action 
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PBEV  Techniques 


*  Measure  quality 

-  Work  products  (partial  and  complete) 

-  Technical  maturity  of  evolving  product 

-  Use  analysis,  models,  simulations,  prototypes 

*  Base  EV  on 

-  Work  products  (drawings,  code)  and 

-  Quality 
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EX  1:  Schedule  Plan  and  Status 


Schedule  Plan 

Jan. 

Feb. 

Mar. 

Apr. 

May 

Total 

Drawings 

8 

10 

12 

10 

10 

50 

Requirements  met: 

Weight 

1 

1 

Diameter 

1 

1 

Status  at  April  30 

*  Drawings  completed:  41 

*  Weight  requirement  not  met 

*  Diameter  requirement  met 
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EX  1:  Earned  Value 


Design 

(drawings) 

Jan. 

Feb. 

Mar. 

Apr. 

May 

Total 

SV  =  - 140 

Planned 
drawings  cur 

8 

10 

12 

10 

10 

50 

Planned 
drawings  cum 

8 

18 

30 

40 

50 

BOWS  cur 

320 

400 

480 

400 

400 

2000 

BCWS  cum 

320 

720 

1200 

1600 

2000  ^ 

2000 

Actual  drawings 
completed  cur 

9 

10 

10 

12 

8 

Actual  drawings 
completed  cum 

9 

19 

29 

41 

49 

EV  (drawings) 
cum 

360 

760 

1160 

1640 

1960 

Negative  EV 

Reqs  cum 

-100 

Net  EV  cum 

360 

760 

1160 

1640  ( 

V £ 

I  00 

lo> 

VQ, 

_ k. 
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EX  1:  Variance  Analysis 


Variance  analysis  (drawings  and  requirements): 

*  1  drawing  behind  schedule  -  40 

*  Diameter  requirement  met  -  0 

*  Weight  requirement  not  met:  - 100 

Schedule  variance 
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Better  Acquisition 
Management 


Solomon 
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Acquisition  Management 


Ensure  Contractors  Integrate  SE  with  EVM 


•  Requirements,  incentives,  insight: 

-  Solicitation/Request  for  Proposal  (RFP) 

-  Integrated  Master  Plan  (IMP) 

-  Integrated  Baseline  Review  (IBR) 

-  Integrated  Master  Schedule  (IMS) 

-  EVMS  compliance  assessments 

-  Independent  technical  assessments 

-  Monitor  consistency  and  validity  of  reports 

-  Independent  EAC  and  risk  assessments 

-  Award  fee  criteria 
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Summary 


*  Integrate 

-  Systems  engineering  with  PBEV 

*  Product  requirements 

*  Manage  the  technical  baseline 

*  Technical  performance  measures 

*  SE  life  cycle  work  products 

-  Technical>schedule>cost  performance 

*  Lean  process 

-  Less  work  packages  with  right  base  measures 

*  Agile 
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Process  I  mprovement 


Using  CMMI®  to  Improve  Earned 
Value  Management 

Paul  Solomon 
October  2002 

Software  Engineering  Process  Management 


Unlimited  distribution  subject  to  the  copyright. 


EARNED  value  management 


Developing  an  EVM  Implementation  Approach 


I 


SOFTWARE  MEASUREMENTS 


Using  Software  Metrics 
& 

Measurements  for  Earned  Value 
Toolkit 


Dave  Burgess  Ted  Rogers 

Cost  Department  Head  EVM  Division  Head 


Chris  Mushrush  Dave  Kester 

EVM  Subject  Matter  Expert  EVM  Subject  Matter  Expert 

October  2004 

Points  of  Contact 

Process:  Earned  Value  Management  AIR  4.2.3 
Technical:  Software  Engineering  AIR  4.1.4 


Sept.  2001 
Aug.  2005 
May  2006 


SEI  /  CMMI  NAVAIR 
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But  wait. 
There’s  more! 

•  Examples 

•  Templates 

•  Tips 

•  Standards 

•  FAR 


Process  I  mprovement 


PERFORMANCE-BASED 
EARNED  VALUE* 


PAUL  J.  SOLOMON.  PMP 
RALPH  R.  YOUNG,  DBA 
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Lessons  Learned  Process 


NDIA  9th  Annual  Systems  Engineering  Conference 

October  24,  2006 

Mr.  Jimmy  Thai 

thaij@saic.com 


Presentation  Outline 


•  The  Issue 

•  The  Challenge 

•  The  7-Step  Solution 

•  Recommendations 

•  Q&A 


2 


From  Science  to  Solutions T 


The  Issue 


♦  Does  any  of  these  problems  sound  familiar  to  your  team? 

•  We  keep  making  the  same  mistakes  over  and  over. 

•  We  thought  someone  on  our  team  knows  how  to  do  this. 

•  Oh  yeah!  We  had  that  problem  with  project  X  too. 

•  Chuck  just  quit  and  there  goes  the  corporate  knowledge. 

♦  Solution:  Having  lessons  learned 

♦  Challenges:  We  are  all  busy,  don’t  have  time,  no  funding,  etc. 


Step  1: 


♦  Everybody  loves  free  lunch 

♦  Attendants  get  instant  benefit 

♦  Do  it  during  lunch  break 

♦  Free  loaders  are  okay 

♦  Benefits  outweigh  cost 


4 


Food 
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Step  2:  Regularly 


♦  Consistence  is  the  key 

♦  The  Pavlov’s  conditioning 

♦  At  least  monthly 

♦  A  habit  needs  21  repetitions 
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Step  3fRelevant  Content 


♦  WIIFM  challenge 

♦  Listen  to  your  audience 

♦  Seek  volunteer  presenters 

♦  Always  have  backup  plan 

♦  Don’t  waste  their  time 
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From  Science  to  Solutions T 


Step  4:  Commitment 


♦  It’s  a  marathon,  not  a  sprint! 

♦  Keep  your  words 

♦  Audience  saw  many  “rah  rah” 
initiatives  before 

♦  Be  resilient 

♦  Send  reminders 
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Step  5T  Part  of  a  Process 


♦  Project  Closeout 

♦  Review  technical,  management, 
financial,  and  people  as  well 

♦  Keep  it  semi-formal 

♦  Bad  news  is  okay 
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Step  6:  Archive 


♦  Not 
everyone 
can  make  it 
today 


♦  Shared  access 
promotes 
collaboration 


♦  Data  ->  Information  ->  Knowledge  ->  Power 
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Step  7:  Application 


♦  Proposal  development  check  list  (i.e.:  Has  the  team  review 
LL?) 

♦  Technical  design  review  check  list 
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Recommendations 


♦  Tailor  to  your  specific  goals 

♦  Start  strong  -  Get  your  ducks  in  a  row 

♦  Keep  cost  low 

♦  Measure  ROI 

♦  Remember  why  you  want  to  do  this  in  the  first  place 


Maritime  Surveillance  &  Security  Division 

Monthly  Technical  Seminar 


*  Want  to  Bo  a  Bettor 
Decision  Maker? 

*  Discover  the  Anatomy 
of  a  Trade  Study 

-  Naii  Your  Solution  with 
Persuasive  Facts 

*  Easily  Document  Your 
Decision 


Which  pizza  is  better,  cheese  or  pepperoni' 


When:  noon  -  1300,  Sep  20,  2006 
Where:  Conference  room  #2309 
Presenter:  Mark  Hogue 
RSVP  by  1500  on  Sep  19 

(Sign-up  s fleet  at  Conference  room's  door) 


fe  * 


Sponsored  by  Systems  Engineering 
Working  Group  (SEIMG) 

POC:  Jimmy  Thai,  *61154 


From  Science  to  Solutions T 


is  is  Your  Conference... 

Please  Ask  Questions 
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m 

invert 

Rapidly  Defining  a 
Lean  CMMI 
Maturity  Level  3 
Process 


Zia  Tufail,  zia@hp.com,  301.233.4228 

Julie  Kellum,  Julie.Kellum@hp.com,  404.731.  52.63 

Tim  Olson-QIC,  Tim.Olson@qic-inc.com,  760.804.1405 


©2004  Hewlett-Packard  Development  Company,  L.P. 

The  information  contained  herein  is  subject  to  change  without  notice 


Presentation  Objectives 


i  n  v  e  n  I 


Describe  CMMI  compliant  online  HP  process 

Outline  Components  for  a  cost  effective,  lean  process 


Describe  HP’s  improvement  objectives. 

Identify  problems  addressed  by  the  streamlined  CMMI  L3  HP 
process 


Describe  the  approach  that  was  used  to  achieve  CMMI 
Maturity  Level  3. 

Unique  techniques  for  defining  &  implementing  an  effective, 
lean  CMMI  L3  process  in  8  months 


Present  some  challenges  and  lessons  learned. 


Answer  any  questions. 


October  26,  2006 
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Outline 


HP  Overview 


HP  Improvement  Objectives 


HP  Approach 


Challenges  and  Lessons  Learne 


Questions  and  Answers 


m 


invert 


October  26,  2006 
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FY05  C&l  EAS/Federal  Initiatives  E3 

invent 

EAS  &  Public  Sector  Initiative  CM  Ml  Relation 


•  CMMI  L3  Achievement 

•  Expand  Application 
Services 

•  Increase  Pursuit 
Capabilities 

•  Delivery  Excellence 

•  Profit  Improvement 


Development  of  EAS  HPGM-AS 
methodology,  rollout  and 
assessment. 

L3  methodology  is  unifying  force  for 
consistent  standards  and  practices. 


CMMI  allows  us  to  pursue 
opportunities  we  would  not  get 
otherwise 

Better  estimation  and  marketing  of 
services 

Standardization  and  enforcement  of 
best  practices. 

Disciplining  delivery  execution  and 
preventing  margin  leakage  by 
fostering  reuse  and  minimizing  risk 


October  26,  2006 
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Existing 
Methods 
Revised  to 
Meet  CMMI 
Level  3 


Pilots 

Underway 


2004 

A  $1.5  Million  dollar,  8  month  project  started  to  revise  Success  Program  W 
meet  CMMI  level  3  and  improve  project  controls.  Some  current  and  new 
FY05  Federal  Contracts  at  risk  without  a  CMMI  level  3  compliant  process. 

•  Sept  CMMI  Level  3  Gap  Analysis  on  Current  Methodology 

•  Oct  -Nov  HPGM-AS  Methodology  Customizations  -  define  per 

Sharepoint  portals 

•  Dec  Pilots  Start : 

•PM  Orientations/Planning  Sessions  per  project 
•CMMI  &  Methodology  Classroom  Training 


2005 

Success  Program  Office  (SEPG  team)  and  Pilot  Projects  start  leveraging 
Success  Program  revised  with  the  new  CMMI  level  3  compliant  process 


Jan 

Feb 

Mar 

April 


Envisioning  &  Maintenance  Phase  Pilots  start 
Design  &  Maintenance  Phase  Pilots 
Build  &  Maintenance  Phase  Pilots 
Pilots  Continue  &  CMMI  Assessment  begins 

April  29  EAS  and  Public  Sector  Achieve  CMMI  level  3  maturity 


October  26,  2006 
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Components  of  the  HP  CMMI  L3  Process 


i  n  v  e  n  I 


Technology 

Sharepoint  based  portals: 
Success  Program  Office 
Success  Manager 
Success  Reviews 
HPGM-AS  Roadmap 
RADM  Virtualization 


Process 

SEI  CMMI  L3  framework 

HP  Global  Method  Application  Services 
(HPGM-AS  -  methodology) 

Success  Reviews  (PPQA  audit  process) 

Success  Manager  (Team  Collaboration 
process) 

Adaptive  Assets  (Knowledge  Capture  & 
Reuse  process) 


Success  Program  Office 
(SEPG  function) 

SQA  Coaches  (per  team) 

Program  Management 

Organization  &  Culture 


People 


Oct 
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Success  Program  Office 


Application  services 
(HPGM-AS) 

•  Delivery  Workflow  portal/process 

•  CMMI  level  3  compliant 

•  Phase  deliverables  &  Signoffs 

Success  Manager 

•  Team  Collaboration  web/process 

•  Pre-Loaded  HPGM-AS 

•  Customer  accessible 

Success  Reviews 

•  Process  Review  (PPQA)  portal/process 

•  Mentoring  Support  By  Field  PMs 

Adaptive  Assets 
Knowledgebase 

Process  and  Tech  Asset  KB  portal/process 

•  Reuse  tool 

Rapid  ADM 

•  Supports  HPGM-AS  Configuration  Mgt 

•  Virtualization  and  Software  Configuration 

•  Management  (SCM)  services 

October  26,  2006 
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1  Home  -  H PGM- AS  Roadmap  R2,6  -  Microsoft  Internet  Explorer  provided  by  Hewlett- Packard 


File  Edit  View  Favorites  Tools  Help 
Q  Back  T  ©•  0  ©  ®|  p  Search 


EME 


ft 


Favorites 


♦ar*-  ■©  0-  &  0  □  fg  ^  ja  0- 


Address  m  https :  // h 20 224.  www2.  hp .  conri/5ites/FrojectsCMMi/SuccessPath_FIPGM-AS_Roadnnaps/Roadmap_v6/default .  aspx 


Go 


Home  Documents  and  Lists  Create  Site  Settings  Help 


Up  to  Success  Manager  HPGM-AS  Roadmaps 


Links 


ft 


HPGM-AS  Roadmap  R2.6 

Home 


This  Site 


Modify  Shared  Page  ^ 


|  Documents 

HPGM-AS  ZIPed 
Templates 

I  Pictures 

Lists 


Second  Release  of  HP  Global  Method  for  Application  Services  which  is  a  level  3  CMMI  application  development: 
methodology  supporting  iterative,  waterfall  and  maintenance  lifecycles  in  HP's  Consulting  and  Integration  (CSd)  area 


HPGM-AS 


New  Projects  X  \ 

START  HERE 

/  Lifecycle  ' 
1  Ptiases- 

Lifecycles 

ENVISIONING 

PHASE 

DESIGN  PHASE 

BUILD  PHASE 

ARTIFACTS 

Project  Planning 
(PP) 

\  HPGM  AS  j 

Project 

/  Protect 

Management 

(PMC) 

1  Support 

Configuration 

Management  (CM) 

Verification  and 

Validation  (VSiV) 

Peer  Review  (VSiV) 

Decision  Analysis  /  \ 

Resolution  (DAR) 

/  SPO 
[  Organkatrart 

Tailoring  (IPM) 

SPO  Success 

Reviews  (PPQA) 

V  Support 

prcieeiTeaift 


Project  Planning  (PF) 


!K  . 


JO 

]  Project  Mgt  Control  (PMC) 

|  Qecfe  ion  Analysis  &  RpsolntfoTi  (OAftj"|  pM 

|  Tailoring  jlPMj  ^ 


1  CofifMjuFation  manage  ment(CM) 

|  Verification  Si  validation  jV&Vj 


Z 


Success  Reviews  (PPQA) 


Measurement  a.  Analysis  (MAj 


Organizational  Process  Improvemeni 
_ jQPD,  flPFj _ 


Organizational  iraniani)  (Of) 


CMMI  Links 

■  SUEMIT  Success  Program 
Improvements  Here! 

■  Adaptive  Assets 

■  Success  Reviews 

■  CMMI  project  webs  (currently  US  PS 
only  -  special  clearance  required) 

■  HPGM-AS  Lifecycles  Overview 

■  SPO  Portal 

■  SPG  Lessons  Learned  Analysis 

■  HPGM-AS  Impact  Analysis  Tool 

■  SPO  Configuration  Management  Plan 

■  SPO  Training  Plan 

■  SPO  Improvement  Plan  (SIP) 

■  SPO  Quality  Policy 

■  SPO  Quality  Assurance  (QAP)  Plan 

■  CMMI  Scope  &  Design  Decisions 


Stcoess  Reviewer 

Q 


SPO  Managers 


Excellance 

Manager 


s  Add  new  link 

Contacts 

~w 

Last  Name 

First  Name 

Role 

Galvan 

Carlos 

April  18-29 

Lead  CMMI 
Assessor 

Erwin 

Erian 

Centers  of 

- 


I  Done 


^  J  Local  intranet 


start 


m  o 


Microsof. . .  t 


Training  20,, . 


HP4  2006 


3  Home 


SEEillEliS# 


October  26,  2006 


Rapid  ADM  Components  and 
Features 


Project  Team  Portal 


Success  Manager 
Pfoject  Team  Porta1  ^ 

are  Configuration  Man*^ 
Complis*^^ 

^fetion  Development  ^o$0) 

2|ng  Model  (SASU 


fOyi 


'sion 


>mated 
ln9  and 


Environment,^} 

Manager 


Rapid  ADM  Development  Platform 


October  26,  2006 
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Outline 


HP  Overview 

HP  Improvement  Objectives 
HP  Approach 
Challenges  and  Lessons  Learne 
Questions  and  Answers 


October  26,  2006 


m 

Success  Program  Vision 

Using  a  sales  and  delivery  integrated  approach,  leverage 
existing,  as  well  as  grow  new  process  and  technical 
Adaptive  Assets  to  : 

Increase  ability  to  compete  in  Federal  bids  through  CMMI  Level  3 
compliance 

Win  more  business  through  improved  differentiation 

•  IBM  differentiation  point 

Increase  Success  of  Projects 

•  Synergistic  Teams 

•  Referencable  Clients 

•  On  Time 

•  In  Budget/Profitable 

•  Effective  HP  Global  Methods  Applied 


October  26,  2006 
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Success  Program  4  What  &  Why 


m 


invent 


Collection  of  specific  process  and  technical  assets  used  in  pre-sales  for 
differentiation  and  in  delivery  to  increase  project  success 


CMMM  Maturity  Level  3  compliant  to  meet  Federal  bid  requirements 


Externally  accessible  outside  HP  Firewall  by  delivery  teams 


•  Major  Areas: 

Success  Path  HPGM-AS 
Success  Manager 
Success  Reviews 
Adaptive  Assets  KB 
Communities  &  Forums 


Best  of  breed  IP  from  HP 


October  26,  2006 
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Outline 


HP  Overview 


HP  Improvement  Objectives 


HP  Approach 


Challenges  and  Lessons  Learne 


Questions  and  Answers 


m 


invert 
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HP  Approach 


i  n  v  e  n  I 


Leverage  existing  HP  processes  and  templates: 

-  Success  Program 

-HP  India  (CMMI  Maturity  Level  5) 

-  HP  Global  Methods  for  Application  Services  (HPGM- 
AS) 

-  HPGM  for  Project  Management  (HPGM-PM) 


Deploy  Tailored  SEI  IDEALSM  Model  and  reuse 
existing  processes  to  be  CMMI  compliant. 


Leverage  Best  of  breed  IP  from  HP 


October  26,  2006 
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SEI  IDEALSM  Model 


m 


invert 


SEI  IDEALsm  Model  for 
Process  Improvement 


Leveraging 


Stimulus  for 
Improvement 

Set  Context 
&  Establish 
Sponsorship 

( Establish  ^ 
Improvement  / 
Infrastructure/ 

1 r  r-r 

,  Appraise  &  \ 

I  IT 

M  u  4  a  4  u  II 

Diagnosing 


Acting 


Establishing 


SM  IDEAL  is  a  service  mark  of  Carnegie  Mellon  University;  Slide  Adapted  from  SEI 


Tailored  SEI IDEALSM  Model  1 

Initiating:  Sponsor  sold  idea  to  HP  senior  management. 


Diagnosing:  HP  India  performed  a  mini-appraisal  (e.g., 
Class  B). 


Establishing:  CMMI  Consultant  established  high-level 
plan  and  HP  established  a  CMMI  Team. 


Acting:  Architected  HP  processes  to  be  CMMI 
compliant  in  2  months;  Skipped  piloting;  Trained  and 
implemented  experienced  projects. 


Success:  Performed  independent  SCAMPI  A. 


26,  2006 
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HP’s  “Lean”  Process 


i  n  v  e  n  I 


HP  Success  Program  is  a  very  “lean”  CMMI 
compliant  process  (about  25%  of  the  size  of 
the  HP  India  process). 


The  process  is  completely  online,  and  uses 
Microsoft  SharePoint. 


HP’s  process  is  only  ~25web  pages  in  size. 


HP  incorporated  best  practices  in  process 
definition 

e.g.,  “Defining  Short,  Usable  Processes  and  Procedures”, 
CrossTalk,  Olson,  Timothy  G.,  June  2006. 


October  26,  2006 
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HPGM-AS  CMMI  L3  Methodology  E3 

^  i  n  v  e  r  I 


2  Paths  depending  on  project 
profile 


Large  HPGM-AS  CMMI 
Implementation 

-  Small  HPGM-AS  CMMI 
Implementation 

Excluded  are  projects 
required  to  use  the 
customer’s  app  dev 
methodology  &  staff  aug 


No  existing  projects  asked 
to  convert  -  new  only 


Large  and  Small  Project 
implementations  differ 
mainly 

•  on  use  of  Success 
Reviews  and 
Deliverables  Required 


l 


Project  Planning  (PP) 


\  Project  Mgt  Control  (PMC)  |  "" 

LDecfe^on^Ana^sis^&^KolutionJDARQ 


PM 


Tailoring  (IPM) 


I 


Tonfjgu^io^management(CM^^] 


JT 


CM /Test  Lead 


Success  Reviews  (PPQA) 


Measurement  &  Analysis  (MA) 

Organizational  Process  Improvement 
_ fOPD,  OPR _ 

Organizational  training  (OT) 


Success  Reviewer 


SPO  Managers 


October  26,  2006 
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IHPGM-AS  Artifact  Comparison 
Example 


Small/Low  Risk 

Large/High  Risk 

Success  Manager  site 

(Issues  List  &  Change  Req/PR  List) 

Success  Manager  site 

(Issues  List  &  Change  Req/PR  List) 

Project  Plan  &  Configuration  Mgt  Plan 

Project  Plan  &  Configuration  Mgt  Plan 

Schedule 

Schedule 

Mini-Spec 

(Contains  sections  from  all  of  these 

Bus  Req  Spec  (BRS) 

Systems  Req  Spec  (SRS) 

Systems  Architecture  (SA) 

Detail  Design  (DD) 

V&V  /Test  Plan 

Req  Traceability  Matrix  (RTM) 

Peer  Review  Log/Checklist 

Mini-Spec 

CM  Audit  Checklists 

Success  Manger,  Product 

Milestone  Review  Phase  Checklists 

Peer  Review  Log/Checklist 

-  Project  plan,  SRS,  SA,  V&V  Plan 

CM  Audit  Checklists 

Success  Manager,  Code,  Product 

Milestone  Review  Phase  Checklists 

October  26,  2006 
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Outline  K3 

HP  Overview 

HP  Improvement  Objectives 
HP  Approach 

Challenges  and  Lessons  Learn 

Questions  and  Answers 


October  26,  2006 
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Challenges 

•  CM  Ml  Resources  were 
reallocated  to  the  new 
high  priorities. 

•  Resistance  to  change 
when  people  have  gone 
through  several  changes 
without  tangible  results. 

•  Aggressive  schedule 

•  Unexpected  events  such 
as  team  health  and 
organizational  changes 


October  26,  2006 
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Lessons  Learned 


i  n  v  e  n  I 


•  It  takes  time,  senior  management  support,  and  mentoring  to 
change  behavior 

•  Traditional  Classroom  training  not  enough  -  critical  to  have  a 
local  resource  to  mentor  team  on  methodology/CMMi  activities 
and  artifacts 

•  Success  Reviews  critical  for  monitoring  and  mentoring  on 
process  use 


•  Critical  to  use  centralized  team  collaboration  tool  for  process 
support  (i.e.  PPQA,  Peer  Reviews,  Lessons  Learned,  Issue 
Tracking/Project  Change  control,  etc) 


Lessons  Learned 


i  n  v  e  n  I 


•  Spend  ample  time  in  planning  phase  to  lock  down  the 
scope  of  the  project. 

•  Obtain  and  maintain  executive  sponsorship  to  keep  driving 
the  project  on  schedule. 

•  Define  Configuration  Management  early  in  the  process. 

•  Follow  the  Configuration  Management  process. 

•  There  is  no  substitute  for  one-on-one  mentoring  for  the  late 
adopters. 

•  Ensure  core  team  understands  the  entire  process. 

•  Ensure  the  process  is  lean  and  relevant. 


Lessons  Learned 


i  n  v  e  n  I 


•  Engage  stakeholders  throughout  the  process. 

•  Third  party  consultants  can  provide  objective 
feedback. 

•  Build  a  diverse  team  and  leverage  diversity. 

•  Ensure  CMMI  expert  resources  are  on  the  team. 

•  CMMI  really  does  require  continuous  process 


improvement. 


What  Went  Well 


i  n  v  e  n  I 


•  Teams  reap  benefits  of  Peer  Reviews,  and 
Milestone  Reviews 

•  Pilot  projects  implement  a  consistent  delivery 
approach 

•  Improved  metrics  to  track  trends  across  projects 

•  Integration  of  virtualization  through  CM  process 

•  Increased  value  to  customer  through  improved 
communications  thru  SPO: 

-  Success  Manager  Team  Webs 

-  Status  Reporting 

-  Automated  Issues  and  Change  Request  tracking 


October  26,  2006 
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What  Went  Well 


i  n  v  e  n  I 


•  Expanded  Rollout  of  Success  program  to  -300 

•  Implemented  new  Success  Program  processes: 

-  HPGM-AS  Bid  Review  (Pre-sales) 

-  HPGM-AS  Quickstarts  (Pre-Delivery,  1  week)  ^  tailoring 
of  the  process  by  SEPG  with  PM  and  SA 

•  Implemented  scalable  training  to  replace 
classroom  training: 

-  PM  CMMI  Training  Track  (14  webinars) 

-  Engineering  CMMI  Training  Track  (9  webinars) 


October  26,  2006 
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Outline 


m 


invert 


HP  Overview 


HP  Improvement  Objectives 


HP  Approach 


Challenges  and  Lessons  Learne 


Questions  and  Answers 
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Questions? 


October  26,  2006 


i  n  v  e  n  I 


Defining  and  Partitioning  Best-of-Team 
Processes  in  a  Multi-Organizational 

Teaming  Environment 


9th  Annual  NDIA  Systems  Engineering  Conference 

October  23-26,  2006 
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Definitions 


•  Process 

•  Organizational  Processes  -the  set  of  processes  that 
defines  what  and  how  the  organization  executes  its 
business 

•  Project  Processes  -  a  tailoring  of  the  organizational 
processes  to  match  the  needs  and  requirements  of  the 
particular  project 

•  Team  -  a  set  of  people  brought  together  for  a  particular 
cause  or  purpose 

•  Multi-Organizational  -  more  than  one  organization 
(relationship  can  be  acquirer/supplier,  team,  joint 
venture,  etc.) 

•  Geographically  Distributed  -  the  team  is  located  in 
more  than  one  physical  location 
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Topics 


•  Definitions  and  Key  Concepts 

•  Examples  of  Multi-Organizational  and 
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USA  Example  of  Business  and  Teaming 
Relationships  -  Future  Combat  Systems 


NH  =4;  CD=2 
MA  =  26;  CD=9 
Rl  =  1 ;  CD=1 
CT  =  4,  CD=2 

NJ  =  25;  CD=9 

MD  =  20,  CD=6 
DC  =  1;  CD=1 


2006 

40  States  &  the  District  of  Columbia 
535  Suppliers 

220  Congressional  Districts 


SYSNC^ATION 


Adapted  from  TACOM,  FCS  Case  06-068 

5 


Systemic  Innovation  for  Business  Results 


European  Example  of  Business  and 
Teaming  Relationships  -  Airbus  A380 


SYSN^TiON 
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Apple  Computer’s  iPod  Global 
Network  of  Innovation 


Ownership,  Branding,  iTunes  -  Apple  Computer 


Original  Idea  -  Tony  Faded,  Independent  Contractor 


Flash  Memory 


Platform  Design  -  PortalPlayer 


1394  Firewire  Interface  Controller  -  Texas  Instruments 


Assembly  &  Packaging  -  Inventec 


-  Sharp  Electronics 
Planar  Lithium  Battery  -  Sony 


Hard  Disk  -  Toshiba 


Stereo  AID  Converter  -  Wolfson  Microelectronics 
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Hypothetical  Teaming  Scenario 


Company 

Size 

Maturity 

Role 

Alpha 

Large 

CMMI  Level  3 

Prime  Contractor 

System  Design  Lead 

Subsystem  2  Lead 

Beta 

Large 

CMMI  Level  5 

Subsystem  1 , 3  Lead 

System  Test  Lead 

Gamma 

Medium 

CMM  Level  5 

SW  Lead  (All  Subsystems) 
System  Integration  Lead 

Delta 

Small 

ISO  9001:2000 

Key  Innovative  Technology 
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So  the  question  is... 


•  Assuming  we  actually  want  to  execute  as  a 
team  (and  not  just  four  separate  companies)... 

•  How  do  we  define  and  partition  the  set  of 
processes  the  team  is  going  to  use? 


Beta 
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Some  Obvious  Challenges 


•  This  project  is  one  of  many  projects  in  each 
organization 

•  Each  organization  has  its  own  culture 

•  Teammates  on  this  project  may  very  well  be 
competitors  on  different  (and  future)  projects 

•  Any  unique  team  processes  must  have  a 
corresponding  infrastructure  and  investment 

•  The  team  needs  to  understand  (through 
communications,  training,  leadership,  etc.)  the 
way  forward 


SYSNC^ATION 


Systemic  Innovation  for  Business  Results 


The  Key  is  Balance... 


The  Other  Extreme: 

All  companies  follow  a  single 
“project”  processes 


One  Extreme: 

All  companies  follow  their 
own  processes 


Balanced  Alternative: 

All  companies  follow  a  combination  of  company 
processes  and  “Best  of  Team  (BoT)”  processes 


(■■■■nnnHHH  »nan*iin4 
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Key  Considerations... 


•  Best  of  Team  (BoT)  Processes  should  be  Determined 
based  on 

•  “Type”  of  process 

•  Policy,  Procedure,  Instruction 

•  “Scope”  of  process 

•  Organizational  vs.  Realization  vs.  Infrastructure 

•  “Level”  of  process 

•  System  vs.  Subsystem  vs.  Component 

•  Strength/maturity/role  of  each  team  member 

•  Also  need  consideration  for 

•  Organizational  Implications 

•  Audits  &  Assessments 
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“Type”  of  Process 
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“Type”  of  Process 


Primary  focus  of  the  BoT  processes. 


For  the  Procedures  that  are 
determined  to  be  BoT,  there  may 
need  to  be  some  common 
Supporting  Materials  leveraged 
as  well.  ****.. 


Supporting  Materials 

Checklists,  Forms,  Templates,  Tools,  Training  Materials 


SYSNOV^TION 
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“Scope”  of  Process 


Org< 


Organizational  Vision  &  Strategy  (OSV) 
Organizational  Quality  &  Improvement  (OQI) 
Business  Development  &  Marketing  (BDM) 


Rea 


Project  Management  (PM) 
Project  Quality  &  Improvement  (PQI) 


Technology  & 
Concept 

Development  (TCD) 


Systems  & 

Services 

Development  (SSD)  ^  Production  (PRD) 


Project  Support  (PS) 


Operations  & 
Support  (OS) 


Disposal  (DSP) 


Infrastructure  Processes 


Communications  (COM) 
Contract  Management  (CON) 
Environmental,  Health  &  Safety  (EHS) 
Ethics  (ETH) 

Export  &  Import  Management  (EIM) 
Facilities  (FAC) 


finance  (FIN) 
l||uman  Resources  (HR) 
Information  Technology  (IT) 
Legal  (LEG) 
Security  (SEC) 
Supplier  Management  (SM) 


SYSNOV^TiON 
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“Scope”  of  Process 


Organizational  Processes 

A 


Realization  Processes 


Organizational  Vision  &  Strategy  (OSV) 
Organizational  Quality  &  Improvement  (OQI) 
Business  Development  &  Marketing  (BDM) 


Project  Management  (PM) 
Project  Quality  &  Improvement  (PQI) 


Technology  & 
Concept 

Development  (TCD) 


Systems  & 

Services 

Development  (SSD)  ^  Production  (PRD) 


Operations  & 
Support  (OS) 


Disposal  (DSP) 


Project  Support  (PS) 


9 

t 


Infrastructure  pr — ss.es  — saiSi 


Communications  (COM) 
Contract  Management  (CON) 
Environmental,  Health  &  Safety  (EHS) 
Ethics  (ETH) 

Export  &  Import  Management  (EIM) 
Facilities  (FAC) 


f  inance  (FIN) 
Human  Resources  (HR) 
Information  Technology  (IT) 
Legal  (LEG) 
Security  (SEC) 
Supplier  Management  (SM) 


sysnc3^tR3n - 


Source:  Sysnovation  Enterprise  Proces^f jajueiMerfc^JEPF)  ©2006  Sysnovation,  LLC 
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The  Development  Process 


Technology 

(Components) 

Adapted  from  ISO/IEC  15288:2002 
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The  Development  Process 


</» 


73 

V) 

7T 


Requirements 


Technology 

(Parts) 


► 


Technology 

(Components) 

Adapted  from  ISO/IEC  15288:2002 
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“Level”  of  Process 


•  Systems  can  be  organized  in  a 
hierarchical  manner 

•  Here  is  a  typical  hierarchical 
definition  where  the  system  is 
designed  from: 

•  supersystem  (if  applicable) 

•  to  system 

•  to  subsystems 

•  to  components 

•  to  subcomponents 

•  to  parts 

•  Typically,  the  system  is 
integrated  in  the  reverse  order 

•  Of  course  there  can  also  be 
special  considerations  such  as 
legacy  systems  and  COTS 
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Supersystems 


Subsystem  Subsystem 


Comp  Comp  Comp  Comp 


Subcomp  Subcomp 


Adapted  from  Kossiakoff  &  Sweet,  1998 
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“Level”  of  Process 


Supersystems 


Subcomp  Subcomp 
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Strength/Maturity/Role  of  Team 


•  Another  key  factor  in  picking  BoT  processes  is  what  each  teammate  has  to  offer 
and  what  their  role  is  on  the  team 


Beta 


•  In  our  example 

•  Alpha  is  the  prime,  but  only  CMMI  Level  3,  they  have  the  lead  on  system 
design,  likely  they  have  solid  project  management  and  subcontracts  processes 
as  well 

•  Beta  is  CMMI  Level  5,  so  perhaps  some  of  their  quantitative  measurement 
processes  would  be  appropriate  for  BoT 

•  Gamma  is  a  SW  house,  so  they  are  likely  to  have  BoT  SW  practices  (but  the 
CMM  Level  5  nature  of  them  may  not  be  “culturally  acceptable”  to  Alpha  and 
Beta) 

•  Delta  is  a  small  outfit  and  is  likely  best  left  to  its  own  processes,  BoT  should  be 
minimized  both  ways 
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Organizational  Implications 


•  Considerations  include: 

•  Process  creation 

•  Process  accessibility/medium 

•  Process  training 

•  Process  measurement  &  effectiveness 

•  Process  improvement 

•  Process  “synchronization” 

•  All  of  these  items  require  investment  and  resources 

•  People 

•  IT  Infrastructure 
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BoT-Based  Audits  &  Assessments 


•  For  example: 

•  ISO  9001:2000  audits 

•  CMMI  assessments 

•  Key  decision: 

•  Rely  on  individual  company’s  evaluations 

•  Perform  audits  &  assessments  on  the  “project” 
processes 

•  Each  has  its  own  set  of  challenges 
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Audits  &  Assessments: 
Individual  Company 


•  Rationale: 

•  Processes  are  core  to  each  company 

•  All  companies  have  numerous  projects 

•  This  project  is  just  another  example  of  tailoring 

•  Challenges 

•  Translation/mapping  of  BoT  processes  to  company 
processes 

•  Training  of  the  practitioners  to  communicate  use  the  BoT 
processes  to  the  evaluators 

•  Dealing  with  the  parts  of  the  BoT  processes  that  are 
done  external  to  your  company  by  your  teammates 
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Audits  &  Assessments: 
Project  Processes 


•  Rationale: 

•  The  BoT  processes  are  unique  to  this  project 

•  One  audit  for  the  team  vs.  separate  audits  for  each  company 

•  Customer  may  require  it 

•  Challenges 

•  May  need  to  bid/budget  for  evaluations  under  project  funds 

•  Extra  expenses  for  separate  evaluations  if  individual 
organization  will  still  be  doing  their  own  evaluations 

•  How  to  define  and  sustain  the  project’s  “organizational” 
processes 

•  Definition,  improvement,  measurement,  evaluation  (separate 
“Project  EPG”  required?) 

•  Training 

•  How  to  leverage  this  project  in  your  other  company  evaluations 


SYSNC^ATION 


Systemic  Innovation  for  Business  Results 
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Summary 


•  Many  projects  are  utilizing  multi-organization, 
geographically  distributed  teams 

•  Special  considerations  need  to  be  made  for 
determining  what  set  of  processes  the  team  will 
be  using. 

•  These  decisions  should  be  made  based  on  the 
type,  scope,  and  level  of  the  processes  and  the 
relative  strengths  and  weaknesses  of  each 
teammate. 
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Comments?  Questions? 
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Disclaimer 


The  views  expressed  are  those  of  the  authors  and 
do  not  reflect  the  official  policy  or  position  of  the 
United  States  Air  Force,  Department  of  Defense, 
United  States  Government,  the  corresponding 
agencies  of  any  other  government,  NATO  or  any 
other  defense  organization. 
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Agenda 


•  Problem  Background 

•  Systems  Engineering  Process 

•  Benefit  Analysis: 

•  Modeling 

•  Simulation 

•  Conclusions 
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Background 


•  Thesis  Proposal 

•  SAF/IA  -  ISHMS  Aging  Aircraft 

•  Constrained  budgets  for  new  aircraft. .  .aircraft  fleets  getting 
older  &  increasing  maintenance  costs 

•  Initial  ISHMS  Goals 

•  Extend  service  life,  maintain  SOF,  &  reduce  maintenance  burden 

-  Reduce  time  to  inspect 

-  Reduce  amount  of  inspection  items 

-  Increase  time  between  inspections 

-  Cost  (cheaper),  schedule  (faster),  performance  (better) 
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Purpose  Statement 


•  Purpose  Statement  -  Two-Fold 

•  To  provide  an  SE  process  to  help  identify  the  top-level  operational 
concept  and  stakeholder  requirements  of  an  ISHMS  for  a  generic 
aging  aircraft 

•  To  perform  a  preliminary  analysis  to  demonstrate  the  potential 
benefit  of  developing  an  ISHMS  for  aging  aircraft 


Stakeholders’  Perspective  and 
Operational  Concept 


•  Life-cycle  Phases 


•  Identify  Stakeholders 


Operational  Concept  Definition 

•  Stakeholders’  vision 

•  Scenarios  for  each  life-cycle  phase 
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Seven-Step  Process 


•  Step  1 :  Develop  Operational  Concept 

•  Step  2:  Define  System  Boundary  With  External  Systems 

•  Step  3:  Develop  Weighted  Objectives  Hierarchy 

•  Step  4:  Develop,  Analyze  And  Refine  Requirements 

•  Step  5:  Ensure  Requirements  Feasibility 

•  Step  6:  Define  Qualification  System  Requirements 

•  Step  7 :  Obtain  Approval  Of  Requirements 


understand  user 
Requirements,  Develop 
System  Concept  and 
Validation  Plan 


Demonstrate  and 
validate  System  to 
User  Validation  Plan 


Develop  System 
Performance  specification 
and  System 
validation  Plan 


integrate  System  and 
Perform  System 

Verification  \q 

Performance  Specification 


Expand  Performance 
Specifications  into  Cl 
Design-to'  Specifications 
and  Cl  Verification  Plan 


Assemble  Cis  and 
Perform  Cl  Verification 
to  Cl  'Design-to’ 
Specrfications 


Evolve  "Design-to 
specifies:  cns  into 
BuikMo*  Documentation 
and  Inspection  Plan 


inspect 

'BuHMnT 

Documentation 


Fan.  Assemble  and 
Code  to  "Build-to’ 
Documentation 


Time 
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DoDAF 

Architecture  Building  Process 


DoDAF  Provides  Architecture  Guidelines 

•  Six-step  Architecture  Building  Process 


G) 


Determine  the 
intended  use  of 
the  architecture 


G) 


Determine  views 
and  products 
to  be  built 


G) 


Determine  the 
scope  of 
the  architecture 


(D 


Gather  data 
and  build  the 
requisite 
products 


G) 


Determine 
characteristics 
to  be  captured 


G) 


Use  architecture 
for  intended 
purpose 
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High-level  Operational  Concept 
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Stakeholders’  Vision 


•  Engineers 


•  Must  deal  with  aging  A-37  problem 

•  Must  decide  whether  to  retire  A-37  aircraft  or  extend  their  life 

•  If  an  extension  is  decided: 


•  How  it  will  be  accomplished? 

•  How  long  the  extension  will  be? 


•  Engineers’  Vision  of  ISHMS 

•  Real-time  recording  of  aircraft  usage  ( loading )  data 

•  Real-time  damage  monitoring 

•  On-board  and  ground  station  analysis  of  recorded  data 

•  Assessment  of  A-37  condition 

•  Predict  remaining  useful  life  of  structure 

•  Development  of  customized  maintenance  schedule 

•  High  reliability 


Their  goal  is  to  make  decisions  on  A-37’s  life  extension  and  maintenance  policy 
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Stakeholders’  Vision 


•  HQ  Maintainers 

•  Must  find  a  solution  to  the  following  problems: 

•  Increased  downtime 

•  Reduced  number  of  operational  aircraft 

•  Extensive  inspections  and  repairs 

•  Inspections  consume  more  resources  (facilities,  equipment,  personnel)  than 


provisioned 


•  Maintainers  at  base  and  squadron  levels  not  able  to  provide  on  time  mission  ready 
aircraft 


•  Maintainers’  Vision  Of  ISHMS 


•  Record  actual  usage  of  the  aircraft 

•  Monitor  structural  damages 

•  Allow  inspections/repairs  to  be  conducted  only  on  the  problem  areas  of  the  aircraft 

•  Reduce  amount  of  teardown  and  overall  maintenance  effort 

•  Inspections  performed  when  required,  not  according  to  a  predetermined  schedule 
(which  may  not  reflect  true  condition  of  aircraft) 


Their  goal  is  to  have  more  efficient  use  of  resources  at  Base  and  Squadron  levels 
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Assumptions 


•  The  ISHMS  will  maintain  or  increase  the  Safety  of  Flight 

•  Maintenance  history  data  available  to  determine  FCLs 

•  Sensor  technology  available  to  monitor  all  of  the  FCLs 

•  Sensor  data  capture  possible  for  all  FCLs 

•  Installation  of  the  sensors  on  the  FCLs  will  not  cause  damage 

•  ISHMS  data  can  be  downloaded  and  analyzed 
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Background/Purpose 


Vision:  CAF  needed  an  integrated  system  that  reduces  the  cost  of  safely  operating 

Cessna  A-37  legacy  aircraft  beyond  design  safe  life 


•  Periodic  Inspection  Intervals  Are  Based  On  Estimates  Of  Fatigue  Crack  Growth  At  FCLs 

•  Safety/Cost  Impacted  If  Inspection  Schedule  Not  Accurate 

•  ISHMS  Optimizes  Inspection/Repair  Cycles. .  .max  Safety  &  Min  Cost 


Estimate  Cost/Safety  Benefit  Of  ISHMS  On  Single  FCL 
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Mech  Theory 


•  Loading...  Stress 


Stress  Amplitude. .  .Fatigue 


F  atigue . . .  Crack  Growth  Rate 


Time 


•  Rate . . .  Cycles  Until  F ailure 
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Design  Objectives 


•  Understand  Aircraft  Reality  FCLs 

•  Create  FE  Simulation  Model 

•  Create  FCL  Max  Stress  Response  Surface  Metamodel 

•  Calculate  Max  Stress  And  Critical  Crack  Length 

•  Calculate  Delta  Stress  Per  Flight  Hour  Cycle  (pfhc) 

•  Delta  Stress  pfhc  And  Critical  Crack  Length  Into 
Walker 

•  Estimate  ISHMS  Benefit  Using  Walker  Crack  Model 


Find  delta  stress  per  flight  hour  cycle. . . 
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Boundary  Conditions 


•  Cantilever  (Fixed-Free) 

•  Clamped  (BCs. .  .x,y,z=0) 


•  Fatigue  Critical  Location 

•  Front  Spar  Wing  Attach  Fitting 

•  Crack  growth  geometry. .  .hole  with  edge  crack 
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Simplifying  Assumptions 


Half-wing  Model 

Simplified  Spar  Geometry  (No  Spar  Caps) 

Interior  Ribs  Not  Meshed  (Stitching) 

Skin  And  Spars  Both  7075-T6  (Verification) 
Elliptical  Lift  Distribution  And  Pressure  Distribution 
Lift  Loading  Isolated  To  Wing  Box 


Simulation  Model 


Response  Surface 
Regression  Metamodel 


Aircraft  Flight  Profile 


FCL  Crack  Growth  Mode! 
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Loading  Conditions 


•  Point  Weapon  Loads 

•  Distributed  SLUF  1  -g  Lift  Load 
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Loading  Conditions 
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Stress  Response  Surface 
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•  Constructed  A  Central  Composite  Design 
For  Front  Spar  Wing  Attach  Fitting 

•  Generated  orthogonal  fractional  factorial  design  to  min  confounding 

•  Determined  design  factor  input  ranges 
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Fighter  Stress  Sequence 
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•  Weighted  General  Fighter  Stress  Sequence 

•  Missions 

•  Weighted  average  effect  on  stress  by  #  of  missions 


Finite  Element 
Simulation;  Model 


I 
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Aircraft  Flight  Profile 
Spectrum 
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FCl  Crack  Growth  Model 
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Benefit  Analysis 
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Analysis 


•  Front  Spar  Wing  Attach  Fitting  Findings: 

•  20,152  psi  stress  per  100  cycles  per  flight  hour 

•  0.4781  inches  critical  crack  length 


Finite  Element 
Simulation  Model 


ZL 

Response  Surface 
Regression  Metannoctel 

i 

Aircraft  Flight  Profile 
Spectrum 

T 

FCL  Crack  Growth  Model 

I 

Benefit  Analysis 
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Simulation  Introduction 


•  Simple  Mathematical  Models 

•  Simulating  crack  growth  over  5000  flight  hours 

•  Single  crack  in  a  single  fatigue  critical  location 

•  Assumed  Damage  Monitoring  Concept 

•  Primary  Variable  -  Time  Between  Scheduled 
Inspections 


Benefit  analysis  to  quantify  the  cost  and  safety  benefits  of  an  ISHMS 
installed  on  an  aged  A-37  aircraft  as  compared  to  the  status  quo  using  simple 

models 
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Simulation  Methodology 


•  Benefit 


•  Failure  rate  per  million  flight  hours  (safety) 

•  Total  number  of  inspections  (cost) 

•  Simulate  Baseline  (No  ISHMS) 

•  Check  model 

•  Establish  benchmark  for  comparison  to  ISHMS 

•  Simulate  With  ISHMS 

•  Began  at  300  flight-hour  interval 

•  Increased  interval 

•  Compared  results  with  Baseline  (percentages) 

•  Sensitivity  Analysis  For  Maintenance  Probability  Of 
Detection 
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Results 


Baseline  -  Failure  vs.  Inspection 


0  100  200  300  400  500  600  700 

Inspection  Interval  (hrs) 


ISHMS  -  Total  Inspections  vs  Inspection  Interval 


ISHMS  -  Failure  Rate  vs.  Inspection  Interval 


Baseline  -  Total  Inspections  vs.  Inspection  Interval 
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Baseline  -  Failure  Rate 


■ 
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Failure  Rate  (per  Min  Fit  Hrs) 


ISHMS  -  Failure  Rate 


ISHMS  -  Failure  Rate  vs.  Inspection  Interval 


Inspection  Interval  (hrs) 


Baseline  -  Inspections 


■ 
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ISHMS  -  Inspections 


■ 
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Simulation  Results 


w 


•  ISHMS  Interval  Selection 

•  Failure  rate  closest  to  Baseline  at  300  flight  hours,  but  no 
worse  than 

•  User  would  ultimately  choose 

•  Comparative  Results 

•  ISHMS  -  700  flight  hours  inspection  interval 

•  Failure  rate  32.8%  lower  (safety) 

•  49.8%  Inspection  Reduction  (cost) 

•  Sensitivity  Analysis 

•  As  probability  of  baseline  crack  detection  increased,  safety 
benefit  decreased 

•  Cost  benefit  stay  relatively  unchanged 
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Thesis  Conclusions 


•  Systems  Engineering  Process 

•  Defining  purpose  sets  the  direction 

•  Operational  Concept  is  critical 

•  Involved  user  is  key 

•  Requirements  are  cornerstone 

•  Architectures  tie  it  all  together 

•  Benefit  Analysis 

•  ISHMS  looks  promising;  need  to  define  its  use  and  design  to 
optimize  benefit 


Starting  off  point  for  future  efforts  in  the  structural  health 

monitoring  field 
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Summary 


•  Problem  Background 

•  Systems  Engineering  Process 

•  Benefit  Analysis: 

•  Modeling 

•  Simulation 

•  Conclusions 
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Questions? 
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Airborne  Warning  and  Control  Systems  Program  Office 


Integrity  -  Service  -  Excellence 


Lessons  Learned  from  a  Mature 
DMSMS  Program 


9th  Annual  Systems  Engineering  Conference 


SQNLDR  P.D. Weeding 

3  Eglin  Street,  Building  1612 
Hanscom  AFB,  MA  01731 
DSN  478-7714 
Commercial  (781)-377-7714 
Fax  (781)-377-1069 


U.S.  AIR  FORCE 


Overview 


U.S.AIR  FORCE 


■  Scope  of  Effort 

■  DMSMS  Goals 

■  Lessons  Learned 

■  Summary 


Integrity  -  Service  -  Excellence 
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Fleet  Facts 


U.S.AIR  FORCE 


■  Design:  Modified  Boeing  707/320 

■  Function:  Airborne  surveillance,  command,  control 
and  communications  (High  Demand) 

■  Fleet  Size:  33  Aircraft  ( Low  Density  -  Small  Fleet) 

■  Age:  30  years  old 

■  Annual  Hours:  18,000 

■  Service  Life:  Extended  to  2035+ 

■  Community  Partners:  NATO  17,  United  Kingdom  7,  Saudi 
Arabia  5,  France  4,  and  Japan  4  Boeing  767  Aircraft 


Integrity  -  Service  -  Excellence 
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Vision  and  Mission 


U.S.AIR  FORCE 


■  Vision  -  “To  never  have  an  AWACS  E-3  become 
mission  incapable  due  to  the  impact  of  DMSMS” 

■  Mission  -  “To  develop  and  implement  an  integrated 
proactive  DMSMS  program  in  support  of  the  AWACS 
Program  Office” 

■  Goal  -  “Drive  down  Total  Ownership  Costs” 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


“ Theoretical ”  Solutions  - 

Cost  and  Time 


ZJ 

CD 

Q_ 

CD 


Data  Source:  DM EA  Cost  Metrics 


Integrity  -  Service  -  Excellence 
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Impact  to  AW  ACS 


U.S.AIR  FORCE 


DMSMS  Impacts  all 
AWACS  commodities 


r 


< 


Air  Frames 

Cockpit,  Fuselage,  Leading 
Edge  Panels  and  Doors 


Mechanical 

Engines,  Hydraulic,  Landing 
Gear  and  Flight  Controls 

Special  Tooling  and  Jigs 


Avionics: 


HF,  VHF,  UHF,  Intercom, 
IFF  and  Emergency  Comm 


AWACS  Unique  Equipment 

Surveillance  Radar,  ESM 
IFF,  etc. 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


Prime  Mission  Equipment 
Obsolescence  over  next  6  years 


Prime  Mission  Equipment 
%  of  Total  Obsolete 


NAV 

14% 


Integrity  -  Service  -  Excellence 
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551  ELSG  DMSMS  Objectives 


U.S.AIR  FORCE 


■  Identify  and  capture  existing  aircraft  configuration 

■  DMSMS  focus  on  current  issues  and  future  mods 

■  Address  present  and  future  supportability  concerns 

■  Preserve  system  and  end-item  OSS&E 

■  Coordinate  and  consolidate  community  efforts 

■  Integrate  and  implement  resolutions  (recommendations) 

■  Mitigate  electronic  and  mechanical  obsolescence 

■  Reduce  both  sustainment  and  development  costs 

■  Institutionalize  proactive  DMSMS  processes 

■  Pinpoint  end-item  and  system  drivers  to  analyze 

■  Track  all  DMSMS  tasks  (WG  database) 


Integrity  -  Service  -  Excellence 


8 


551st  DMSMS  Process  Flow 


U.S.AIR  FORCE 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


Operational  Impact 
Analysis  (OIA)* 
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*  The  OIA  is  an  ARINC 
Proprietary  Tool 


Integrity  -  Service  -  Excellence 
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AW  ACS  Priority  Database 


U.S.AIR  FORCE 


SQL  Database 

Stores 

Calculates 

Exports 

Schedules 


*  The  Priority  Database  is  an 
ARINC  Proprietary  Tool 


Integrity  -  Service  -  Excellence 
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DMSMS  Approach 


U.S.AIR  FORCE 


SRU 


Drill  down 

assembling  Component 
and  analyzing 
Logistics  and 
Engineering 
data  along  the 
entire  path 


SE  Review 


Summary  Report 


Problem  Part  Report 


Roll  up  of  all  data  points 
including  resolution  with 
costs  comparisons 


Applicable  drawing  and  its  revision 

Type  of  drawing  and  item  reviewed 

Configuration  and  design 

Next  higher  assembly  and  end  item  applicability 

Potential  resolutions  to  the  obsolete  components 


Engineering 

Authority 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


551st  ELSG  Data  Repository 


Aircraft/System  application 


Technical  characteristics 
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Integrity  -  Service  -  Excellence 
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ALARM©  Shapshot 


U.S.AIR  FORCE 


■  ARINC  Process:  Assess  obsolescence  characteristics  of  each  component  in  an  end  item 
BOM,  assign  risk  rating,  roll  up  to  LRU  rating,  and  rank  the  LRU  relative  to  other  LRUs: 


LRU  Individual  Risk 
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Component  is  displayed 


LRU  Rank  Order  by 
Composite  Risk 

Can  be  reported  by: 

•  Specified  LRU 

•  Top  25,  50, 100 

•  Type  (electronic  vs. 
Mechanical) 

•  By  organization  within 
specified  MDS 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


ALARM©  Shapshot 


ARINC  Process 


Identify  “High”  risk  component  items  and  perform  engineering  assessment  to 
determine  unique  component  characteristics  contributing  to  obsolescence  risk: 

Technical  data  completeness 
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ALARM©  Shapshot 


U.S.AIR  FORCE 


■  ARINC  Process 

Create  mechanical  attribute  library 
Data  availability 

Applicable  specifications/standards  and  their  status 
Tool  availability 
Banned  chemicals 
Sources  no  bid 


OticiqHkMclAimUDI  HEADING  GYRDS-1DPE  SET 


Technical  Data  |  M.-Jt-ir*  and  Piece- l:  as  |  T  --.i  f .1/1  btf:  j  Chefiacats  [  |  Rill: 

~7J  CwiO 


tt  p 

zi 

Flev  Dale;]i.1  DevtJJ? 

Ftfr[Miiil  rwy  [?E 

zn 

Drawing  CAGE] 

D  e  e  hgn  Ac  tlvidC  y ■  | 

~n 

Drmiig  Tjew:  |MIL-SPEC 

j 

Diawintj  -  | 

D  al  l  L"ij**tp|p1r.  j  --ESi 

“23 

epical 
LNmmiiotii. 


& 


►  |SQCK£T  plA^TEf  -  | 

I  Ok-  MAX  Ltl  MIN  -  | 

Add 

Qdele 

Sj’iwrr 

.u _ i  >r 

m  mT 

mn 

Sri 

A.W 

1 

«]  _ 

1 

•1 

She-  Or-mrig  KbcX  is  nsctire  end  was  laplaccd  by  dm  nurber  Alp^h^i-  'iftSi’ TitS  dated  flFjfftL/203] 
iun#0*  . 


Unaxiigti  Di-.iwimjfl 


Data  stored  based  on  drawing, 
allows  multiple  PNs  within 
drawing  to  be  documented  and 
retained 

PN  specific  data  stored  separately 

As  library  grows  for  each  category 
of  data,  less  information  to 
manually  document 
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Lessons  Learned 


U.S.AIR  FORCE 


■  VHF  radio 

■  One  of  the  first  systems  on  which  an  evaluation  a 
DMSMS  was  developed 

■  Assumed  configuration  in  predictive  health  tool  was  correct 

■  Performed  assessment  even  though  OIA  showed  no  support 
impacts  up  to  2024 

■  Did  not  research  if  system  was  to  be  replaced 

■  Learned  configuration  in  predictive  health  tool  did  not 
include  all  SRUs  (Over  20  CCAs  not  assessed) 

■  Researched  drawing  data  to  extract  correct  configuration 


■  Found  out  system  was  scheduled  to  be  replaced  in  2010 

■  Changed  process  to  include  system  manager  in 
determining  if  DMSMS  assessment  required  and  when 
system  was  scheduled  for  replacement 
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Lessons  Learned 


U.S.AIR  FORCE 


■  Standardized  Central  Air  Data  Computer  (SCADC)  and  UHF  Radio 

■  USAF  program  offices  and  FMS  customers  concerned  that  a 
new  modified  SCADC  would  not  be  supportable 

■  UHF  radio  DMSMS  issues  raised  by  FMS  community 

■  DMSMS  assessments  initiated 

■  OlAs  for  the  SCADC  and  UHF  radio  showed  that  no  obsolescence  impact  to 
mission  support  due  to  large  number  of  spares  in  the  supply  system 

■  SCADC  removed  from  decommissioned  fleet  (C-141 ,  2  per),  and 
spares  placed  in  inventory  (over  400) 

■  UHF  radio  inventory  build  up  due  to  replacement  of  radio  on  some 
fleets 

■  Determined  that  even  though  obsolescence  may  adversely 
impact  SRUs: 

■  If  enough  LRU  spares  (serviceable  and  reparable)  exist,  obsolescence 
impacts  are  negated  because  cannibalization  can  support  extended 
requirements 

■  Use  preliminary  OIA  to  determine  which  LRUs  to  assess  (if  green 
through  2026  or  green  past  replacement  date,  defer  analysis 


■  Must  take  supply  chain  into  account  versus  only 
obsolescence  data 
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Lessons  Learned 


U.S.AIR  FORCE 


■  Large  Systems 

■  Radar 

■  Over  NNN  LRUs  in  the  radar  system 

■  Top  NN  obsolescence  LRUs  in  the  radar 

■  Inundated  the  engineering  authority  for  the  radar  with  proposed 
resolutions  to  obsolescence 

■  Decision  made  to  alternate  between  radar  and  other 
system  LRUs 

■  Otherwise  could  have  spent  entire  contract  period  on  a 
single  system 

■  Need  to  show  DMSMS  working  for  all  stakeholders 
in  the  your  community 
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Lessons  Learned 


U.S.AIR  FORCE 


■  Data  key  to  success 

■  Configuration  data 

■  Technical  Data 

■  Market  Surveys  for  COTS 

■  Teaming  Required 

■  MOAs  mapped  out  and  agreed  upon 

■  WG  responsibilities  defined 

■  Due  to  differing  DMSMS  methodologies  across  user  community 

■  Return  on  Investment  (ROI) 

■  None  experienced  without  implementation  of  solutions 

■  POM  justification  can  be  generated  in  proactive  program 

■  DMSMS  focus  on  current  issues  and  future  mods 

■  Data  points 

■  Just  a  starting  point... no  one  tool  is  enough 

■  Prioritization... assessment  of  Supply  Chain  can  focus  resources 

■  Focus  on  areas  with  greatest  impact 
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Lessons  Learned 


U.S.AIR  FORCE _ 

■  Proactive... Cost  Avoidance  growth  per  year 

■  2003-  $823,500 

■  Resolutions- 153 

■  ROI-  4.1  to  1 

■  2004-  $2.56M 

■  Resolutions-  354 

■  ROI-  5.6  to  1 

■  2005-  $3.80M 

■  Resolutions- 184 

■  ROI- 16.1  to  1 


To  date  2006-  $2.96M 
Resolutions- 146 
ROI- 15.7  to  1 
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U.S.AIR  FORCE 


QUESTIONS 
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Agenda 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


>  Defining  Commonality  Requirements  for  FCS 

-  The  Purpose  of  Commonality 

-  Commonality  Definition 

-  Commonality  Requirements  Allocation 

-  Status  of  Commonality  on  FCS 

•  Considerations  in  Pursuing  Commonality 

•  Measuring  Commonality  Implementation 

•  Summary 

•  Questions 
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FUTURE  COMBAT  SYSTEMS 


FCS  System-of-Systems  (SoS) 


One  Tem-  IbeAtmy/Defoase/lndusiry 


FCS  Recovery  and 
Maintenance 
Vehicle  (FRMV) 


ical  Vehicle 
Evacuation  (MV-E) 


Medical 
Treatment  (MV-T) 


Unmanned  Ground  Vehicles 


ARV  RSTA 


Small 

(Manpackable)  UGV 


ARV 


Armed  Robotic 
Vehicle  (ARV) 


ARV-A  (L) 


aft 

MULE  MULE 

(Countermine)  (Transport)8'1'06 


Non-Line  of 
Cannon  (N 


Mortar  (NLOS-M) 


Unattended  Munitions 

NLOS  L 


Unattended 
Ground 
Sensors  ■ 


siLOS  LS  Intelligent 

Munitions 
Systems 


Class  II 


Class  I 


System^ 


2  v  - * 


ommand  and 


Control  Vehicle  (C2V) 


Mounted  Combat 
System  (MCS) 


and 

Nance  Vehicle  (RSV) 


Infantry  Carrier 
Vehicle  (ICV) 


Unmanned 


Aerial  Vehicles 
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Commonality  Requirement 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


The  System-of-Systems  shall  maximize  platform  and 
component  commonality  within  each  system  class 

to  a  level  of  70%. 


•  The  Problem: 

-  What  does  the  requirement  mean  ...  at  a  SoS  Level? 

-  How  does  the  requirement  apply  to  individual  systems? 

-  What  does  it  mean  to  be  “common”? 

-  How  do  you  measure/verify  performance? 

•  This  requirement  interacts  with  other  requirements  for  common 
tools,  common  lubricants,  common  batteries,  commonality  in 
communications  equipment,  and  others. 

Need  to  Translate  the  User’s  Requirement  into  Engineering  Language 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Commonality  Focus 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


Commonality  in  the  Factory 

-  Decrease  production  cost 

•  Fewer  unique  tasks 

•  Reduced  training  requirements 

•  Reduced  manufacturing  complexity 

-  Streamline  supply  chain 

Commonality  in  the  Field 

-  Simplify  sustainment 

-  Reduce  sustainment 

-  Support  multi-functionality 

-  Reduce  personnel  requirements 

-  Reduce  training  requirements 


User  Focus  is  Commonality  in  the  Field 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Why  Commonality? 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


•  Achieve  Efficiencies  in  Maintainability 

-  Fewer  unique  installations 

•  Fewer  unique  maintenance/repair  procedures 

-  Reduces  training  requirements 

-  Requires  fewer  unique  skills 

-  Accelerates  repair/replacement 

•  Achieve  Efficiencies  in  Operational  Availability 

-  Less  down  time  due  to 

•  Faster  repair/replacement  of  failed  components 

•  Reduced  spares  requirements  /  Less  dependency  on  the  supply  chain 

•  Ability  to  swap  like  components  between  systems  to  meet  mission  needs 

•  Achieve  Efficiencies  in  Life  Cycle  Cost 

-  Fewer  unique  parts 

-  Reduced  spares  requirements 

-  Reduced  training  requirements 

Commonality  is  a  Key  Ingredient  to  Meeting  the  FCS  Program 
Objective  to  Reduce  Logistics  Footprint 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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What  does  it  mean  to  be  “common”? 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


A  part  is  considered  “common”  if  the  following  criteria  are 
met: 

(a)  The  part  is  designated  as  a  field-replaceable  part 

(b)  The  part  is  functionally  required  on  multiple  systems  in  a  specific 
class 

(c)  The  part,  with  the  same  NSN  number,  is  equally  qualified  for  use 
on  all  systems  in  a  specific  class  without  modification. 

Common  Component:  A  field  replaceable  LRU/LRM  that  is 
used  in  the  same  application  on  multiple  systems  in  a 
system  class,  and  has  the  same  NSN  number  regardless  of 
system  application. 

Establish  a  Common  Set  of  Expectations 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Definition  of  the  System  Classes 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


For  the  purposes  of  component  commonality,  the  systems  classes  are 
defined  as  follows: 


Class 

Systems  Included 

Commonality  Requirement 

MGV 

All 

Applies 

MULE 

MULE-T,  MULE-C,  ARV-A(L) 

Applies 

ARV 

ARV-A,  ARV-RSTA 

Applies 

SUGV 

Self 

Does  Not  Apply 

Class  1  UAV 

Self 

Does  Not  Apply 

Class  2  UAV 

Self 

Does  Not  Apply 

Class  3  UAV 

Self 

Does  Not  Apply 

Class  4  UAV 

Self 

Does  Not  Apply 

T-UGS 

Self 

Does  Not  Apply 

U-UGS 

Self 

Does  Not  Apply 

NLOS-LS 

Self 

Does  Not  Apply 

IMS 

Self 

Does  Not  Apply 

Apply  Commonality  Where  it  Makes  Sense 


Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Requirement  Development 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


User  Requirement 

The  System -of -Systems  shall  maximize  platform  and  component 
commonality  within  each  system  class  to  a  level  of  70%. 

Svstem-of-Svstems  Design  Requirement 

The  FCS  Platforms  shall  use  common  LRUs/LRMs  to  allow  for  70% 
interchangeability  within  the  classes  for  each  major  FCS  system. 

Svstem/Platform  Design  Requirement 

The  PRIME  ITEM  shall  have  70%  common  and  interchangeable 
LRUs/LRMs  within  each  class. 


Commonality  Definitions  Facilitate  Allocation  of  Meaningful 
Requirements  at  Multiple  Levels  in  the  FCS  Product  Structure 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Status  of  Commonality  on  FCS 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


•  Component  commonality  is  a  contractual  program  requirement,  specified 
in  the  ORD  and  Statement  of  Work  (SOW) 

•  Commonality  requirements  are  defined  in  the  System-of-Systems 
Specification 

•  Commonality  requirements  are  included  in  the  Prime  Item  Development 
Specifications  (PIDS)  which  establish  the  requirements  baseline  for  each 
FCS  system 

•  Incorporation  of  component  commonality  into  the  system  design  is 
required  in  each  One  Team  Partner’s  SOW 

•  Expectations  for  component  commonality  across  the  SoS  are  clearly 
defined 


The  FCS  Program  is  Committed  to  Achieving 
Component  Commonality  Objectives 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Agenda 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


^Defining  Commonality  Requirements  for  FCS 

>  Considerations  in  Pursuing  Commonality 

-  Identifying  Common  Functionality 

-  Program  Set-up 

•  Measuring  Commonality 

•  Summary 

•  Questions 


Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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FUTURE  COMBAT  SYSTEMS 


Considerations  in  Pursuing  Commonality 


One  Jem-  TheAmy/Defonsefindusiry 


•  Establish  commonality  objectives  where  it  makes  sense, 
taking  into  consideration: 

-System  functionality 

-  System  performance 

-  System  maintenance  concepts 

-  Life  Cycle  Cost 

*  The  opportunity  for  commonality  is  greatest  between 
systems  that  have  common  or  similar  functionality 

-  Identify  common  functions  between  multiple  systems 

-  Focus  commonality  objectives  on  the  systems/subsystems  that 
perform  similar  functions 

-Adapt  definition  and  allocation  of  functions  within  the  system 
architecture  to  facilitate  the  use  of  common  components 

Expectations  for  commonality  need  to  be  realistic 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Identifying  Common  Functions  -  SoS  Level 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAtmy/Detefx&ftfidustfy 


•  Identify  the  Systems  Classes 

•  Identify  common  functions  between  systems  =  where  it  would  be  logical  to 
have  common  components 


Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Identifying  Common  Functions  -  System  Level 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAmy/Detens&ftfidustfy 


Identifying  Common  Functions  -  System  Level 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 
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Program  Setup 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


•  Commonality  objectives  integrated  early  in  the  development 
effort 

-  Program  goals  and  objectives 

-  Identification  of  the  system  functionality  and  performance 

-  Definition  of  the  product  structure 

•  Commonality  is  more  than  a  requirements  issue 

-  Concept  of  Operations 

•  Commonality  needs  to  be  driven  by  user  goals  and  objectives 

-  Business  Management 

•  Commonality  objectives  need  to  be  part  of  the  program  business  model 

-  Supplier  Management 

•  Suppliers  need  to  be  contractually  obligated  and  incentivized  to  achieve 
commonality  objectives 

Commonality  Needs  to  be  Built  Into  the  Basic  Foundations 

of  a  Program 

Approved  for  Public  Release,  Distribution  Unlimited,  TACOM  19  SEP  2006,  case  06-202. 
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Agenda 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


^Defining  Commonality  Requirements  for  FCS 
s Considerations  in  Pursuing  Commonality 
>  Measuring  Commonality 

•  Summary 

•  Questions 
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Compliance  with  Commonality 
Requirements 


fUWRS  COMBAT  SYSTEMS 


•  Early  attempts  on  FCS  to  define  an  methodology  for  determining 
compliance  with  the  commonality  requirements  looked  at  statistical 
analysis 

Resulted  from  many  different  interpretations  of  what  it  meant  to  be  common  and 
how  it  applied  to  the  FCS  systems 

Lacked  relevance  -  difficult  to  relate  back  to  desired  design  characteristics  and 


performance 


•  Establishment  of  the  commonality  definitions  facilitated  discussions  on 
how  to  count  common  versus  unique  LRUs/LRMs 

-  Perceived  as  a  more  tangible  assessment  of  compliance 

-  Directly  related  to  the  system  design 

-  Discussed  as  Approach  1  on  the  next  chart 

•  Consideration  of  commonality  as  a  SoS  requirement  being  addressed  in 
a  system  design/development  environment  leads  to  the  proposal  of 
Approach  2 

-  Potential  for  more  realistic  expectations 

Allows  for  consistent  assessment  between  system  and  SoS  perspectives 


Commonality  Approaches  are  Still  Evolving 
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Measuring  Commonality  -  Approach  1 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


Commonality  Requirement  (CR) 


Total  Number  of  LRUs/LRMs  that  should  be  common 
Total  Number  of  LRUs/LRMs 


Total  Number  of  LRUs/LRMs  that  are  common 

Commonality  Performance  (Cp)  =  - ^  — - - , ,  - 

3  v  p/  Total  Number  of  LRUs/LRMs 


System  1 

14  Parts 
10  LRUs/LRMs  (Bold) 

5  LRUs/  LRMs  should  be  common 
4  LRUs/  LRMs  are  common 


16  LRUs/ LRMs  (Bold) 

5  LRUs/  LRMs  should  be  common 
4  LRUs/  LRMs  are  common 


•  Each  box  is  a  part 

•  Bold  indicates  a  part  that  has 
been  designated  as  an 
LRU/LRM 

•  ■  I  ndicates  an  LRU/LRM  that 
should  be  common 

•  Of  the  LRUs/LRMs  that  should 
be  common,  only  the  ones  in 
the  intersection  actually  are 
common 


10  LRUs/ LRMs  (Bold) 

4  LRUs/  LRMs  should  be  common 
4  LRUs/  LRMs  are  common 


CR  =  5/ 10  =  50  % 
CP  =  4/ 10  =  40  % 


CR  =  5/  16  =  31  % 
CP  =  4/  16  =  25  % 


CR  =  4/  10  =  40  % 
CP  =  4/  10  =  40  % 
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Measuring  Commonality  -  Approach  2 


FUTURE  COMBAT  SYSTEMS 


One  Tern-  TheAimy/Detens&ftfidustfy 


Total  Number  of  LRUs/LRMs  that  could  be  common 

Commonality  Potential  (CJ  =  - t  .  ,  K, — - . - 

3  v  p/  Total  Number  of  LRUs/LRMs 


Commonality  Ratio  (Cr)  = 


Total  Number  of  LRUs/LRMs  that  are  common 
Total  Number  of  LRUs/LRMs  that  could  be  common 


System  1 

14  Parts 
10  LRUs/LRMs  (Bold) 

5  LRUs/  LRMs  could  be  common 
4  LRUs/  LRMs  are  common 


•  Each  box  is  a  part 

•  Bold  indicates  a  part  that  has 
been  designated  as  an 
LRU/LRM 

•  ■  I  ndicates  an  LRU/LRM  that 
should  be  common 

•  Of  the  LRUs/LRMs  that  should 
be  common,  only  the  ones  in 
the  intersection  actually  are 
common 


16  LRUs/ LRMs  (Bold) 

5  LRUs/  LRMs  could  be  common 
4  LRUs/  LRMs  are  common 


10  LRUs/ LRMs  (Bold) 

4  LRUs/  LRMs  could  be  common 
4  LRUs/  LRMs  are  common 


Cp  =  5/  10  =  .5  Cp  =  5/  16  =  .3  Cp  =  4/  10  =  .4 

Cr  =  4/  5  =  .8  Cr  =  4/  5  =  .8  Cr  =  4/  4  =  1 
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*  Approach  1 

-  Percentages,  as  a  definition  of  the  level  of  commonality  to  be  exhibited 
by  a  system  to  be  designed,  do  not  translate  well  into  design 
requirements 

-  Percentages,  as  an  assessment  of  performance  or  compliance,  can  be 
misleading 

*  Approach  2 

-A  measure  of  “Commonality  Potential”,  based  on  the  evaluation  of 
common  functions  in  a  conceptual  design,  can  provide  more  realistic 
expectations  for  a  specific  development  effort 

-A  “Commonality  Ratio”  provides  an  assessment  of  commonality  that  is 
more  consistent  across  multiple  systems  in  a  SoS  environment 

Assessment  Approach  Needs  to  be  Developed  Concurrently 

with  Definitions  and  Requirements 
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•  FCS  is  actively  working  to  meet  the  program  commonality 
objectives 

•  The  FCS  program  SoS  approach  to  commonality  is  based  upon: 

-  A  clear  definition  of  Commonality 

•  What  is  Commonality? 

•  What  is  the  purpose  of  Commonality  on  a  specific  program? 

•  How  are  Commonality  criteria  applied  to  the  system  architecture? 

-  Commonality  objectives  included  in  the  initial  formulation  of  a  program 

-  Commonality  objectives  synchronized  with  the  system  architecture  and 
conceptual  design 

•  Measures  of  achievement  of  commonality  objectives  are  being 
developed  using  the  same  definitions  and  expectations  that 
established  the  commonality  requirements 
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